Nov 01 05:00:20.153166 ip-10-0-129-92 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Nov 01 05:00:20.153176 ip-10-0-129-92 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Nov 01 05:00:20.153183 ip-10-0-129-92 systemd[1]: kubelet.service: Failed with result 'resources'.
Nov 01 05:00:20.153400 ip-10-0-129-92 systemd[1]: Failed to start Kubernetes Kubelet.
Nov 01 05:00:30.196036 ip-10-0-129-92 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Nov 01 05:00:30.196051 ip-10-0-129-92 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 73ecc3921f88480cadc1253394410c4e --
Nov 01 05:02:27.710799 ip-10-0-129-92 systemd[1]: Starting Kubernetes Kubelet...
Nov 01 05:02:28.076859 ip-10-0-129-92 kubenswrapper[2571]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 05:02:28.076859 ip-10-0-129-92 kubenswrapper[2571]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 01 05:02:28.076859 ip-10-0-129-92 kubenswrapper[2571]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 05:02:28.077793 ip-10-0-129-92 kubenswrapper[2571]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 05:02:28.081244 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081170    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 05:02:28.081284 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081246    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 05:02:28.081284 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081250    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 05:02:28.081284 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081253    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 05:02:28.081284 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081256    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 05:02:28.081284 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081259    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 05:02:28.081284 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081262    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 05:02:28.081284 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081272    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 05:02:28.081284 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081278    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 05:02:28.081284 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081282    2571 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 05:02:28.081284 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081286    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 05:02:28.081284 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081289    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 05:02:28.081594 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081292    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 05:02:28.081594 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081294    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 05:02:28.081594 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081297    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 05:02:28.081594 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081300    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 05:02:28.081594 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081302    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 05:02:28.081594 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081305    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 05:02:28.081594 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081308    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 05:02:28.081594 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081310    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 05:02:28.081594 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081313    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 05:02:28.081594 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081315    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 05:02:28.081594 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081318    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 05:02:28.081594 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081321    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 05:02:28.081594 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081323    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 05:02:28.081594 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081326    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 05:02:28.081594 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081328    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 05:02:28.081594 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081331    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 05:02:28.081594 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081334    2571 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 05:02:28.081594 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081336    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 05:02:28.081594 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081338    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 05:02:28.082038 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081341    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 05:02:28.082038 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081343    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 05:02:28.082038 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081346    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 05:02:28.082038 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081348    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 05:02:28.082038 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081351    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 05:02:28.082038 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081353    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 05:02:28.082038 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081355    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 05:02:28.082038 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081358    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 05:02:28.082038 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081360    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 05:02:28.082038 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081364    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 05:02:28.082038 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081366    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 05:02:28.082038 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081369    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 05:02:28.082038 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081372    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 05:02:28.082038 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081375    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 05:02:28.082038 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081378    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 05:02:28.082038 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081380    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 05:02:28.082038 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081383    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 05:02:28.082038 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081385    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 05:02:28.082038 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081388    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 05:02:28.082038 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081390    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 05:02:28.082521 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081393    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 05:02:28.082521 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081395    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 05:02:28.082521 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081397    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 05:02:28.082521 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081400    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 05:02:28.082521 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081402    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 05:02:28.082521 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081404    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 05:02:28.082521 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081407    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 05:02:28.082521 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081409    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 05:02:28.082521 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081412    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 05:02:28.082521 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081414    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 05:02:28.082521 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081417    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 05:02:28.082521 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081419    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 05:02:28.082521 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081422    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 05:02:28.082521 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081425    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 05:02:28.082521 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081427    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 05:02:28.082521 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081430    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 05:02:28.082521 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081432    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 05:02:28.082521 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081435    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 05:02:28.082521 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081437    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 05:02:28.082952 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081439    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 05:02:28.082952 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081443    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 05:02:28.082952 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081447    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 05:02:28.082952 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081468    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 05:02:28.082952 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081471    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 05:02:28.082952 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081474    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 05:02:28.082952 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081477    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 05:02:28.082952 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081480    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 05:02:28.082952 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081483    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 05:02:28.082952 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081485    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 05:02:28.082952 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081488    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 05:02:28.082952 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081490    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 05:02:28.082952 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081493    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 05:02:28.082952 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081495    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 05:02:28.082952 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081498    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 05:02:28.082952 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081500    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 05:02:28.082952 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081503    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 05:02:28.082952 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081505    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 05:02:28.082952 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081508    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 05:02:28.082952 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081511    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 05:02:28.083407 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081513    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 05:02:28.083407 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081516    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 05:02:28.083407 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081518    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 05:02:28.083407 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081522    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 05:02:28.083407 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081524    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 05:02:28.083407 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081527    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 05:02:28.083407 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081529    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 05:02:28.083407 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081532    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 05:02:28.083407 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081534    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 05:02:28.083407 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081537    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 05:02:28.083407 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081539    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 05:02:28.083407 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081542    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 05:02:28.083407 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081544    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 05:02:28.083407 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081546    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 05:02:28.083407 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081549    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 05:02:28.083407 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.081551    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 05:02:28.083407 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082309    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 05:02:28.083407 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082316    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 05:02:28.083407 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082319    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 05:02:28.083407 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082322    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 05:02:28.083890 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082325    2571 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 05:02:28.083890 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082328    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 05:02:28.083890 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082330    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 05:02:28.083890 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082333    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 05:02:28.083890 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082335    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 05:02:28.083890 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082338    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 05:02:28.083890 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082341    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 05:02:28.083890 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082343    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 05:02:28.083890 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082345    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 05:02:28.083890 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082348    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 05:02:28.083890 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082350    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 05:02:28.083890 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082353    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 05:02:28.083890 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082355    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 05:02:28.083890 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082358    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 05:02:28.083890 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082360    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 05:02:28.083890 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082363    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 05:02:28.083890 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082365    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 05:02:28.083890 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082368    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 05:02:28.083890 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082370    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 05:02:28.083890 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082373    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 05:02:28.084555 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082376    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 05:02:28.084555 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082379    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 05:02:28.084555 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082381    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 05:02:28.084555 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082384    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 05:02:28.084555 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082386    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 05:02:28.084555 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082389    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 05:02:28.084555 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082392    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 05:02:28.084555 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082394    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 05:02:28.084555 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082396    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 05:02:28.084555 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082407    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 05:02:28.084555 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082410    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 05:02:28.084555 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082413    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 05:02:28.084555 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082416    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 05:02:28.084555 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082418    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 05:02:28.084555 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082420    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 05:02:28.084555 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082423    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 05:02:28.084555 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082425    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 05:02:28.084555 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082428    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 05:02:28.084555 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082431    2571 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 05:02:28.085262 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082434    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 05:02:28.085262 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082436    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 05:02:28.085262 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082439    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 05:02:28.085262 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082441    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 05:02:28.085262 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082444    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 05:02:28.085262 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082446    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 05:02:28.085262 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082464    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 05:02:28.085262 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082467    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 05:02:28.085262 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082470    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 05:02:28.085262 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082472    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 05:02:28.085262 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082475    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 05:02:28.085262 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082477    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 05:02:28.085262 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082480    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 05:02:28.085262 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082483    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 05:02:28.085262 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082486    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 05:02:28.085262 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082489    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 05:02:28.085262 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082491    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 05:02:28.085262 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082494    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 05:02:28.085262 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082496    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 05:02:28.085937 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082499    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 05:02:28.085937 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082501    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 05:02:28.085937 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082503    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 05:02:28.085937 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082506    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 05:02:28.085937 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082509    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 05:02:28.085937 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082513    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 05:02:28.085937 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082517    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 05:02:28.085937 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082521    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 05:02:28.085937 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082524    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 05:02:28.085937 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082527    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 05:02:28.085937 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082530    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 05:02:28.085937 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082532    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 05:02:28.085937 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082535    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 05:02:28.085937 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082537    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 05:02:28.085937 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082540    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 05:02:28.085937 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082543    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 05:02:28.085937 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082545    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 05:02:28.085937 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082548    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 05:02:28.085937 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082550    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 05:02:28.086394 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082553    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 05:02:28.086394 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082555    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 05:02:28.086394 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082560    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 05:02:28.086394 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082563    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 05:02:28.086394 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082566    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 05:02:28.086394 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082569    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 05:02:28.086394 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082572    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 05:02:28.086394 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082574    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 05:02:28.086394 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082577    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 05:02:28.086394 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082579    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 05:02:28.086394 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082582    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 05:02:28.086394 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082585    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 05:02:28.086394 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082587    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 05:02:28.086394 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082590    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 05:02:28.086394 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082592    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 05:02:28.086394 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082595    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 05:02:28.086394 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082597    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 05:02:28.086394 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082600    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 05:02:28.086394 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082603    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 05:02:28.086876 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082606    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 05:02:28.086876 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082609    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 05:02:28.086876 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082611    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 05:02:28.086876 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082614    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 05:02:28.086876 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082616    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 05:02:28.086876 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.082619    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 05:02:28.086876 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083148    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 05:02:28.086876 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083154    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 05:02:28.086876 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083157    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 05:02:28.086876 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083160    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 05:02:28.086876 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083163    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 05:02:28.086876 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083165    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 05:02:28.086876 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083168    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 05:02:28.086876 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083170    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 05:02:28.086876 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083173    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 05:02:28.086876 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083176    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 05:02:28.086876 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083178    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 05:02:28.086876 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083181    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 05:02:28.086876 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083183    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 05:02:28.086876 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083186    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 05:02:28.087347 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083189    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 05:02:28.087347 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083191    2571 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 05:02:28.087347 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083194    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 05:02:28.087347 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083197    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 05:02:28.087347 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083199    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 05:02:28.087347 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083203    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 05:02:28.087347 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083207    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 05:02:28.087347 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083210    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 05:02:28.087347 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083212    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 05:02:28.087347 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083215    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 05:02:28.087347 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083217    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 05:02:28.087347 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083220    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 05:02:28.087347 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083223    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 05:02:28.087347 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083225    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 05:02:28.087347 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083228    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 05:02:28.087347 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083230    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 05:02:28.087347 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083233    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 05:02:28.087347 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083235    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 05:02:28.087347 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083238    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 05:02:28.087915 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083240    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 05:02:28.087915 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083242    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 05:02:28.087915 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083245    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 05:02:28.087915 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083247    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 05:02:28.087915 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083250    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 05:02:28.087915 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083252    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 05:02:28.087915 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083255    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 05:02:28.087915 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083257    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 05:02:28.087915 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083260    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 05:02:28.087915 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083262    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 05:02:28.087915 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083265    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 05:02:28.087915 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083268    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 05:02:28.087915 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083270    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 05:02:28.087915 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083274    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 05:02:28.087915 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083276    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 05:02:28.087915 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083279    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 05:02:28.087915 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083281    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 05:02:28.087915 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083285    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 05:02:28.087915 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083288    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 05:02:28.088354 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083291    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 05:02:28.088354 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083293    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 05:02:28.088354 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083296    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 05:02:28.088354 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083298    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 05:02:28.088354 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083300    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 05:02:28.088354 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083303    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 05:02:28.088354 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083306    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 05:02:28.088354 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083308    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 05:02:28.088354 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083311    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 05:02:28.088354 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083313    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 05:02:28.088354 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083316    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 05:02:28.088354 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083318    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 05:02:28.088354 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083321    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 05:02:28.088354 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083324    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 05:02:28.088354 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083326    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 05:02:28.088354 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083328    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 05:02:28.088354 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083331    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 05:02:28.088354 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083333    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 05:02:28.088354 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083336    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 05:02:28.088354 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083339    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 05:02:28.088848 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083341    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 05:02:28.088848 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083344    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 05:02:28.088848 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083346    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 05:02:28.088848 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083349    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 05:02:28.088848 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083351    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 05:02:28.088848 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083354    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 05:02:28.088848 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083359    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 05:02:28.088848 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083361    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 05:02:28.088848 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083364    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 05:02:28.088848 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083366    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 05:02:28.088848 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083368    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 05:02:28.088848 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083371    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 05:02:28.088848 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083373    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 05:02:28.088848 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083376    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 05:02:28.088848 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083378    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 05:02:28.088848 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083380    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 05:02:28.088848 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083383    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 05:02:28.088848 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083385    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 05:02:28.088848 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083388    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 05:02:28.089285 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083390    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 05:02:28.089285 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083393    2571 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 05:02:28.089285 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083395    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 05:02:28.089285 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083398    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 05:02:28.089285 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083400    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 05:02:28.089285 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083403    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 05:02:28.089285 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083405    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 05:02:28.089285 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083408    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 05:02:28.089285 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083410    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 05:02:28.089285 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083413    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 05:02:28.089285 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083415    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 05:02:28.089285 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083418    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 05:02:28.089285 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083421    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 05:02:28.089285 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083423    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 05:02:28.089285 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.083426    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 05:02:28.089285 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083523    2571 flags.go:64] FLAG: --address="0.0.0.0"
Nov 01 05:02:28.089285 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083530    2571 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 01 05:02:28.089285 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083536    2571 flags.go:64] FLAG: --anonymous-auth="true"
Nov 01 05:02:28.089285 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083542    2571 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 01 05:02:28.089285 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083549    2571 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 01 05:02:28.089285 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083554    2571 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 01 05:02:28.089817 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083560    2571 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 01 05:02:28.089817 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083567    2571 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 01 05:02:28.089817 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083571    2571 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 01 05:02:28.089817 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083573    2571 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 01 05:02:28.089817 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083577    2571 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 01 05:02:28.089817 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083580    2571 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 01 05:02:28.089817 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083583    2571 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 01 05:02:28.089817 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083586    2571 flags.go:64] FLAG: --cgroup-root=""
Nov 01 05:02:28.089817 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083589    2571 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 01 05:02:28.089817 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083591    2571 flags.go:64] FLAG: --client-ca-file=""
Nov 01 05:02:28.089817 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083594    2571 flags.go:64] FLAG: --cloud-provider="external"
Nov 01 05:02:28.089817 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083598    2571 flags.go:64] FLAG: --cluster-dns="[]"
Nov 01 05:02:28.089817 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083605    2571 flags.go:64] FLAG: --cluster-domain=""
Nov 01 05:02:28.089817 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083609    2571 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 01 05:02:28.089817 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083613    2571 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 01 05:02:28.089817 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083616    2571 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 01 05:02:28.089817 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083619    2571 flags.go:64] FLAG: --container-log-max-files="5"
Nov 01 05:02:28.089817 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083623    2571 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 01 05:02:28.089817 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083627    2571 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 01 05:02:28.089817 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083630    2571 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 01 05:02:28.089817 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083633    2571 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 01 05:02:28.089817 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083636    2571 flags.go:64] FLAG: --contention-profiling="false"
Nov 01 05:02:28.089817 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083642    2571 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 01 05:02:28.089817 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083645    2571 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 01 05:02:28.090419 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083648    2571 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 01 05:02:28.090419 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083651    2571 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 01 05:02:28.090419 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083656    2571 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 01 05:02:28.090419 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083660    2571 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 01 05:02:28.090419 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083665    2571 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 01 05:02:28.090419 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083668    2571 flags.go:64] FLAG: --enable-load-reader="false"
Nov 01 05:02:28.090419 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083672    2571 flags.go:64] FLAG: --enable-server="true"
Nov 01 05:02:28.090419 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083675    2571 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 01 05:02:28.090419 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083678    2571 flags.go:64] FLAG: --event-burst="100"
Nov 01 05:02:28.090419 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083682    2571 flags.go:64] FLAG: --event-qps="50"
Nov 01 05:02:28.090419 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083685    2571 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 01 05:02:28.090419 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083688    2571 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 01 05:02:28.090419 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083691    2571 flags.go:64] FLAG: --eviction-hard=""
Nov 01 05:02:28.090419 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083695    2571 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 01 05:02:28.090419 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083698    2571 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 01 05:02:28.090419 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083701    2571 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 01 05:02:28.090419 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083704    2571 flags.go:64] FLAG: --eviction-soft=""
Nov 01 05:02:28.090419 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083707    2571 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 01 05:02:28.090419 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083710    2571 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 01 05:02:28.090419 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083717    2571 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 01 05:02:28.090419 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083721    2571 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 01 05:02:28.090419 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083724    2571 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 01 05:02:28.090419 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083727    2571 flags.go:64] FLAG: --fail-swap-on="true"
Nov 01 05:02:28.090419 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083730    2571 flags.go:64] FLAG: --feature-gates=""
Nov 01 05:02:28.090419 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083734    2571 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 01 05:02:28.091029 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083737    2571 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 01 05:02:28.091029 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083740    2571 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 01 05:02:28.091029 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083743    2571 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 01 05:02:28.091029 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083746    2571 flags.go:64] FLAG: --healthz-port="10248"
Nov 01 05:02:28.091029 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083748    2571 flags.go:64] FLAG: --help="false"
Nov 01 05:02:28.091029 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083753    2571 flags.go:64] FLAG: --hostname-override="ip-10-0-129-92.us-west-1.compute.internal"
Nov 01 05:02:28.091029 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083756    2571 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 01 05:02:28.091029 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083759    2571 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 01 05:02:28.091029 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083761    2571 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 01 05:02:28.091029 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083764    2571 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 01 05:02:28.091029 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083768    2571 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 01 05:02:28.091029 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083770    2571 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 01 05:02:28.091029 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083773    2571 flags.go:64] FLAG: --image-service-endpoint=""
Nov 01 05:02:28.091029 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083776    2571 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 01 05:02:28.091029 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083779    2571 flags.go:64] FLAG: --kube-api-burst="100"
Nov 01 05:02:28.091029 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083782    2571 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 01 05:02:28.091029 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083785    2571 flags.go:64] FLAG: --kube-api-qps="50"
Nov 01 05:02:28.091029 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083787    2571 flags.go:64] FLAG: --kube-reserved=""
Nov 01 05:02:28.091029 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083790    2571 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 01 05:02:28.091029 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083793    2571 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 01 05:02:28.091029 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083796    2571 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 01 05:02:28.091029 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083799    2571 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 01 05:02:28.091029 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083801    2571 flags.go:64] FLAG: --lock-file=""
Nov 01 05:02:28.091592 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083804    2571 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 01 05:02:28.091592 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083807    2571 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 01 05:02:28.091592 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083810    2571 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 01 05:02:28.091592 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083815    2571 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 01 05:02:28.091592 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083818    2571 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 01 05:02:28.091592 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083821    2571 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 01 05:02:28.091592 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083824    2571 flags.go:64] FLAG: --logging-format="text"
Nov 01 05:02:28.091592 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083827    2571 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 01 05:02:28.091592 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083830    2571 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 01 05:02:28.091592 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083833    2571 flags.go:64] FLAG: --manifest-url=""
Nov 01 05:02:28.091592 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083836    2571 flags.go:64] FLAG: --manifest-url-header=""
Nov 01 05:02:28.091592 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083840    2571 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 01 05:02:28.091592 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083843    2571 flags.go:64] FLAG: --max-open-files="1000000"
Nov 01 05:02:28.091592 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083846    2571 flags.go:64] FLAG: --max-pods="110"
Nov 01 05:02:28.091592 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083851    2571 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 01 05:02:28.091592 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083854    2571 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 01 05:02:28.091592 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083857    2571 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 01 05:02:28.091592 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083860    2571 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 01 05:02:28.091592 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083862    2571 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 01 05:02:28.091592 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083865    2571 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 01 05:02:28.091592 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083868    2571 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 01 05:02:28.091592 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083876    2571 flags.go:64] FLAG: --node-status-max-images="50"
Nov 01 05:02:28.091592 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083879    2571 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 01 05:02:28.091592 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083882    2571 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 01 05:02:28.091592 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083885    2571 flags.go:64] FLAG: --pod-cidr=""
Nov 01 05:02:28.092198 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083888    2571 flags.go:64] FLAG: --pod-manifest-path=""
Nov 01 05:02:28.092198 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083890    2571 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 01 05:02:28.092198 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083894    2571 flags.go:64] FLAG: --pods-per-core="0"
Nov 01 05:02:28.092198 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083896    2571 flags.go:64] FLAG: --port="10250"
Nov 01 05:02:28.092198 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083899    2571 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 01 05:02:28.092198 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083902    2571 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0fdeb5f686df6389c"
Nov 01 05:02:28.092198 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083905    2571 flags.go:64] FLAG: --qos-reserved=""
Nov 01 05:02:28.092198 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083908    2571 flags.go:64] FLAG: --read-only-port="10255"
Nov 01 05:02:28.092198 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083911    2571 flags.go:64] FLAG: --register-node="true"
Nov 01 05:02:28.092198 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083913    2571 flags.go:64] FLAG: --register-with-taints=""
Nov 01 05:02:28.092198 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083917    2571 flags.go:64] FLAG: --registry-burst="10"
Nov 01 05:02:28.092198 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083920    2571 flags.go:64] FLAG: --registry-qps="5"
Nov 01 05:02:28.092198 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083923    2571 flags.go:64] FLAG: --reserved-cpus=""
Nov 01 05:02:28.092198 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083925    2571 flags.go:64] FLAG: --reserved-memory=""
Nov 01 05:02:28.092198 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083929    2571 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 01 05:02:28.092198 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083932    2571 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 01 05:02:28.092198 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083935    2571 flags.go:64] FLAG: --rotate-certificates="false"
Nov 01 05:02:28.092198 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083938    2571 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 01 05:02:28.092198 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083941    2571 flags.go:64] FLAG: --runonce="false"
Nov 01 05:02:28.092198 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083943    2571 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 01 05:02:28.092198 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083950    2571 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 01 05:02:28.092198 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083955    2571 flags.go:64] FLAG: --seccomp-default="false"
Nov 01 05:02:28.092198 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083957    2571 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 01 05:02:28.092198 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083960    2571 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 01 05:02:28.092198 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083963    2571 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 01 05:02:28.092198 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083966    2571 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 01 05:02:28.092810 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083969    2571 flags.go:64] FLAG: --storage-driver-password="root"
Nov 01 05:02:28.092810 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083972    2571 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 01 05:02:28.092810 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083978    2571 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 01 05:02:28.092810 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083980    2571 flags.go:64] FLAG: --storage-driver-user="root"
Nov 01 05:02:28.092810 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083983    2571 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 01 05:02:28.092810 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083986    2571 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 01 05:02:28.092810 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083989    2571 flags.go:64] FLAG: --system-cgroups=""
Nov 01 05:02:28.092810 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083991    2571 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 01 05:02:28.092810 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083997    2571 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 01 05:02:28.092810 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.083999    2571 flags.go:64] FLAG: --tls-cert-file=""
Nov 01 05:02:28.092810 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.084002    2571 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 01 05:02:28.092810 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.084005    2571 flags.go:64] FLAG: --tls-min-version=""
Nov 01 05:02:28.092810 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.084008    2571 flags.go:64] FLAG: --tls-private-key-file=""
Nov 01 05:02:28.092810 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.084011    2571 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 01 05:02:28.092810 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.084013    2571 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 01 05:02:28.092810 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.084016    2571 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 01 05:02:28.092810 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.084019    2571 flags.go:64] FLAG: --v="2"
Nov 01 05:02:28.092810 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.084023    2571 flags.go:64] FLAG: --version="false"
Nov 01 05:02:28.092810 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.084027    2571 flags.go:64] FLAG: --vmodule=""
Nov 01 05:02:28.092810 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.084031    2571 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 01 05:02:28.092810 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.084034    2571 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 01 05:02:28.092810 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085126    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 05:02:28.092810 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085141    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 05:02:28.092810 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085147    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 05:02:28.093391 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085151    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 05:02:28.093391 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085162    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 05:02:28.093391 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085168    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 05:02:28.093391 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085175    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 05:02:28.093391 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085179    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 05:02:28.093391 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085186    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 05:02:28.093391 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085194    2571 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 05:02:28.093391 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085199    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 05:02:28.093391 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085204    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 05:02:28.093391 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085209    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 05:02:28.093391 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085215    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 05:02:28.093391 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085219    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 05:02:28.093391 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085224    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 05:02:28.093391 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085233    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 05:02:28.093391 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085237    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 05:02:28.093391 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085242    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 05:02:28.093391 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085246    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 05:02:28.093391 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085250    2571 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 05:02:28.093391 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085254    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 05:02:28.093857 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085259    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 05:02:28.093857 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085263    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 05:02:28.093857 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085267    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 05:02:28.093857 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085272    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 05:02:28.093857 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085276    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 05:02:28.093857 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085280    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 05:02:28.093857 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085285    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 05:02:28.093857 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085293    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 05:02:28.093857 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085297    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 05:02:28.093857 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085301    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 05:02:28.093857 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085306    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 05:02:28.093857 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085311    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 05:02:28.093857 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085315    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 05:02:28.093857 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085319    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 05:02:28.093857 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085324    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 05:02:28.093857 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085328    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 05:02:28.093857 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085333    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 05:02:28.093857 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085337    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 05:02:28.093857 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085341    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 05:02:28.093857 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085345    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 05:02:28.094323 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085355    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 05:02:28.094323 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085359    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 05:02:28.094323 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085364    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 05:02:28.094323 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085370    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 05:02:28.094323 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085375    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 05:02:28.094323 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085379    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 05:02:28.094323 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085383    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 05:02:28.094323 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085387    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 05:02:28.094323 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085391    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 05:02:28.094323 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085395    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 05:02:28.094323 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085400    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 05:02:28.094323 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085403    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 05:02:28.094323 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085412    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 05:02:28.094323 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085416    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 05:02:28.094323 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085420    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 05:02:28.094323 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085424    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 05:02:28.094323 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085428    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 05:02:28.094323 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085433    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 05:02:28.094323 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085438    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 05:02:28.094776 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085442    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 05:02:28.094776 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085446    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 05:02:28.094776 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085470    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 05:02:28.094776 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085476    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 05:02:28.094776 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085481    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 05:02:28.094776 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085484    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 05:02:28.094776 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085488    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 05:02:28.094776 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085493    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 05:02:28.094776 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085497    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 05:02:28.094776 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085501    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 05:02:28.094776 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085506    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 05:02:28.094776 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085510    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 05:02:28.094776 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085514    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 05:02:28.094776 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085518    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 05:02:28.094776 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085527    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 05:02:28.094776 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085531    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 05:02:28.094776 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085537    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 05:02:28.094776 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085544    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 05:02:28.094776 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085550    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 05:02:28.094776 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085555    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 05:02:28.095242 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085560    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 05:02:28.095242 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085563    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 05:02:28.095242 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085567    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 05:02:28.095242 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085571    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 05:02:28.095242 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085575    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 05:02:28.095242 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085579    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 05:02:28.095242 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085584    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 05:02:28.095242 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085592    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 05:02:28.095242 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085596    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 05:02:28.095242 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085600    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 05:02:28.095242 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085605    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 05:02:28.095242 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085609    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 05:02:28.095242 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085613    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 05:02:28.095242 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085617    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 05:02:28.095242 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085621    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 05:02:28.095242 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085626    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 05:02:28.095242 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085630    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 05:02:28.095242 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085634    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 05:02:28.095242 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085638    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 05:02:28.095702 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085642    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 05:02:28.095702 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085650    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 05:02:28.095702 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085654    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 05:02:28.095702 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085658    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 05:02:28.095702 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085663    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 05:02:28.095702 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085668    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 05:02:28.095702 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.085915    2571 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 05:02:28.095702 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085972    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 05:02:28.095702 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085979    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 05:02:28.095702 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085983    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 05:02:28.095702 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085986    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 05:02:28.095702 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085990    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 05:02:28.095702 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085993    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 05:02:28.095702 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085996    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 05:02:28.095702 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.085999    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 05:02:28.096053 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086001    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 05:02:28.096053 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086006    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 05:02:28.096053 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086008    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 05:02:28.096053 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086011    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 05:02:28.096053 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086014    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 05:02:28.096053 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086016    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 05:02:28.096053 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086019    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 05:02:28.096053 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086022    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 05:02:28.096053 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086024    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 05:02:28.096053 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086029    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 05:02:28.096053 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086031    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 05:02:28.096053 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086034    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 05:02:28.096053 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086037    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 05:02:28.096053 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086039    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 05:02:28.096053 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086042    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 05:02:28.096053 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086045    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 05:02:28.096053 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086047    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 05:02:28.096053 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086050    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 05:02:28.096053 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086052    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 05:02:28.096053 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086055    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 05:02:28.096526 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086057    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 05:02:28.096526 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086060    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 05:02:28.096526 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086062    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 05:02:28.096526 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086064    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 05:02:28.096526 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086068    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 05:02:28.096526 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086071    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 05:02:28.096526 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086075    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 05:02:28.096526 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086079    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 05:02:28.096526 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086082    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 05:02:28.096526 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086085    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 05:02:28.096526 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086087    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 05:02:28.096526 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086090    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 05:02:28.096526 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086092    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 05:02:28.096526 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086095    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 05:02:28.096526 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086097    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 05:02:28.096526 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086100    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 05:02:28.096526 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086102    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 05:02:28.096526 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086105    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 05:02:28.096526 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086107    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 05:02:28.096526 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086110    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 05:02:28.096985 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086112    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 05:02:28.096985 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086115    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 05:02:28.096985 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086118    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 05:02:28.096985 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086120    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 05:02:28.096985 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086123    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 05:02:28.096985 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086125    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 05:02:28.096985 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086128    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 05:02:28.096985 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086130    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 05:02:28.096985 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086133    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 05:02:28.096985 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086136    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 05:02:28.096985 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086139    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 05:02:28.096985 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086142    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 05:02:28.096985 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086144    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 05:02:28.096985 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086147    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 05:02:28.096985 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086149    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 05:02:28.096985 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086152    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 05:02:28.096985 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086155    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 05:02:28.096985 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086158    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 05:02:28.096985 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086160    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 05:02:28.097424 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086163    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 05:02:28.097424 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086166    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 05:02:28.097424 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086168    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 05:02:28.097424 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086172    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 05:02:28.097424 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086176    2571 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 05:02:28.097424 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086179    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 05:02:28.097424 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086181    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 05:02:28.097424 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086184    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 05:02:28.097424 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086187    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 05:02:28.097424 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086190    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 05:02:28.097424 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086192    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 05:02:28.097424 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086195    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 05:02:28.097424 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086197    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 05:02:28.097424 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086200    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 05:02:28.097424 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086202    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 05:02:28.097424 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086205    2571 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 05:02:28.097424 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086207    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 05:02:28.097424 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086210    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 05:02:28.097424 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086213    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 05:02:28.097424 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086215    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 05:02:28.097919 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086217    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 05:02:28.097919 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086220    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 05:02:28.097919 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086222    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 05:02:28.097919 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086224    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 05:02:28.097919 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086227    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 05:02:28.097919 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086231    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 05:02:28.097919 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086234    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 05:02:28.097919 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086237    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 05:02:28.097919 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086239    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 05:02:28.097919 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086245    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 05:02:28.097919 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086247    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 05:02:28.097919 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086250    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 05:02:28.097919 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086253    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 05:02:28.097919 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086255    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 05:02:28.097919 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086258    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 05:02:28.097919 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086260    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 05:02:28.097919 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086263    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 05:02:28.097919 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086265    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 05:02:28.097919 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.086269    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 05:02:28.097919 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.093437    2571 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 01 05:02:28.098403 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.093465    2571 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 01 05:02:28.098403 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.093480    2571 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 01 05:02:28.098403 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.093485    2571 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 01 05:02:28.098403 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093553    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 05:02:28.098403 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093558    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 05:02:28.098403 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093561    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 05:02:28.098403 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093564    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 05:02:28.098403 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093566    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 05:02:28.098403 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093569    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 05:02:28.098403 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093572    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 05:02:28.098403 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093574    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 05:02:28.098403 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093577    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 05:02:28.098403 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093579    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 05:02:28.098403 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093582    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 05:02:28.098403 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093584    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 05:02:28.098403 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093587    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 05:02:28.098403 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093589    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 05:02:28.098403 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093592    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 05:02:28.098403 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093594    2571 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 05:02:28.098881 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093597    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 05:02:28.098881 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093599    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 05:02:28.098881 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093602    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 05:02:28.098881 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093605    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 05:02:28.098881 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093607    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 05:02:28.098881 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093610    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 05:02:28.098881 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093613    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 05:02:28.098881 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093615    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 05:02:28.098881 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093617    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 05:02:28.098881 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093620    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 05:02:28.098881 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093622    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 05:02:28.098881 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093624    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 05:02:28.098881 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093627    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 05:02:28.098881 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093632    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 05:02:28.098881 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093635    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 05:02:28.098881 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093637    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 05:02:28.098881 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093640    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 05:02:28.098881 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093642    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 05:02:28.098881 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093645    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 05:02:28.099334 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093648    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 05:02:28.099334 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093650    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 05:02:28.099334 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093653    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 05:02:28.099334 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093655    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 05:02:28.099334 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093657    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 05:02:28.099334 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093660    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 05:02:28.099334 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093662    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 05:02:28.099334 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093665    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 05:02:28.099334 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093667    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 05:02:28.099334 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093670    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 05:02:28.099334 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093672    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 05:02:28.099334 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093675    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 05:02:28.099334 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093677    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 05:02:28.099334 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093680    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 05:02:28.099334 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093682    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 05:02:28.099334 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093685    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 05:02:28.099334 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093687    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 05:02:28.099334 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093689    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 05:02:28.099334 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093692    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 05:02:28.099334 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093694    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 05:02:28.099907 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093696    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 05:02:28.099907 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093707    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 05:02:28.099907 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093710    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 05:02:28.099907 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093713    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 05:02:28.099907 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093716    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 05:02:28.099907 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093718    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 05:02:28.099907 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093721    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 05:02:28.099907 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093725    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 05:02:28.099907 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093728    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 05:02:28.099907 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093730    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 05:02:28.099907 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093733    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 05:02:28.099907 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093735    2571 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 05:02:28.099907 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093738    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 05:02:28.099907 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093740    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 05:02:28.099907 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093743    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 05:02:28.099907 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093745    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 05:02:28.099907 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093748    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 05:02:28.099907 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093750    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 05:02:28.099907 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093753    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 05:02:28.099907 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093755    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 05:02:28.100395 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093758    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 05:02:28.100395 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093760    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 05:02:28.100395 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093763    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 05:02:28.100395 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093765    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 05:02:28.100395 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093767    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 05:02:28.100395 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093770    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 05:02:28.100395 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093772    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 05:02:28.100395 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093775    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 05:02:28.100395 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093777    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 05:02:28.100395 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093779    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 05:02:28.100395 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093782    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 05:02:28.100395 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093784    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 05:02:28.100395 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093787    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 05:02:28.100395 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093790    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 05:02:28.100395 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093792    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 05:02:28.100395 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093794    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 05:02:28.100395 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093797    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 05:02:28.100395 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093799    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 05:02:28.100395 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093802    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 05:02:28.100859 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093806    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 05:02:28.100859 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093811    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 05:02:28.100859 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093814    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 05:02:28.100859 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093818    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 05:02:28.100859 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093821    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 05:02:28.100859 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093823    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 05:02:28.100859 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093826    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 05:02:28.100859 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093829    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 05:02:28.100859 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093831    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 05:02:28.100859 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093833    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 05:02:28.100859 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093836    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 05:02:28.100859 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.093838    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 05:02:28.100859 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.093866    2571 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 05:02:28.100859 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094009    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 05:02:28.100859 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094014    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 05:02:28.101193 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094017    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 05:02:28.101193 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094019    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 05:02:28.101193 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094022    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 05:02:28.101193 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094024    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 05:02:28.101193 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094027    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 05:02:28.101193 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094029    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 05:02:28.101193 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094032    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 05:02:28.101193 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094034    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 05:02:28.101193 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094037    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 05:02:28.101193 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094039    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 05:02:28.101193 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094042    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 05:02:28.101193 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094044    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 05:02:28.101193 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094047    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 05:02:28.101193 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094049    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 05:02:28.101193 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094052    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 05:02:28.101193 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094054    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 05:02:28.101193 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094056    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 05:02:28.101193 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094059    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 05:02:28.101193 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094062    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 05:02:28.101649 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094068    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 05:02:28.101649 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094071    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 05:02:28.101649 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094073    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 05:02:28.101649 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094076    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 05:02:28.101649 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094078    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 05:02:28.101649 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094080    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 05:02:28.101649 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094083    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 05:02:28.101649 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094085    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 05:02:28.101649 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094088    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 05:02:28.101649 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094090    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 05:02:28.101649 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094093    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 05:02:28.101649 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094095    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 05:02:28.101649 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094098    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 05:02:28.101649 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094100    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 05:02:28.101649 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094102    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 05:02:28.101649 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094105    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 05:02:28.101649 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094107    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 05:02:28.101649 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094110    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 05:02:28.101649 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094112    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 05:02:28.101649 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094115    2571 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 05:02:28.102151 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094117    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 05:02:28.102151 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094120    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 05:02:28.102151 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094122    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 05:02:28.102151 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094125    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 05:02:28.102151 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094127    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 05:02:28.102151 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094130    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 05:02:28.102151 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094132    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 05:02:28.102151 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094135    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 05:02:28.102151 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094137    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 05:02:28.102151 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094140    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 05:02:28.102151 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094142    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 05:02:28.102151 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094145    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 05:02:28.102151 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094147    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 05:02:28.102151 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094151    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 05:02:28.102151 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094153    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 05:02:28.102151 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094156    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 05:02:28.102151 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094158    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 05:02:28.102151 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094160    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 05:02:28.102151 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094163    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 05:02:28.102151 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094165    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 05:02:28.102619 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094168    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 05:02:28.102619 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094170    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 05:02:28.102619 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094172    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 05:02:28.102619 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094174    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 05:02:28.102619 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094177    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 05:02:28.102619 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094181    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 05:02:28.102619 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094184    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 05:02:28.102619 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094187    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 05:02:28.102619 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094190    2571 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 05:02:28.102619 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094192    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 05:02:28.102619 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094195    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 05:02:28.102619 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094197    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 05:02:28.102619 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094200    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 05:02:28.102619 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094202    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 05:02:28.102619 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094204    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 05:02:28.102619 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094207    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 05:02:28.102619 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094209    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 05:02:28.102619 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094211    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 05:02:28.102619 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094214    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 05:02:28.103166 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094217    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 05:02:28.103166 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094219    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 05:02:28.103166 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094221    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 05:02:28.103166 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094224    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 05:02:28.103166 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094226    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 05:02:28.103166 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094228    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 05:02:28.103166 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094231    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 05:02:28.103166 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094233    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 05:02:28.103166 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094237    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 05:02:28.103166 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094241    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 05:02:28.103166 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094243    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 05:02:28.103166 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094246    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 05:02:28.103166 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094248    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 05:02:28.103166 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094250    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 05:02:28.103166 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094253    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 05:02:28.103166 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094255    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 05:02:28.103166 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094258    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 05:02:28.103166 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094260    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 05:02:28.103166 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094263    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 05:02:28.103642 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094265    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 05:02:28.103642 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094268    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 05:02:28.103642 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094270    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 05:02:28.103642 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094272    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 05:02:28.103642 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094277    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 05:02:28.103642 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094280    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 05:02:28.103642 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094282    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 05:02:28.103642 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.094304    2571 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 05:02:28.103642 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094336    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 05:02:28.103642 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094340    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 05:02:28.103642 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094342    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 05:02:28.103642 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094345    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 05:02:28.103642 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094347    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 05:02:28.103642 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094350    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 05:02:28.103642 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094354    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 05:02:28.104023 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094356    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 05:02:28.104023 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094359    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 05:02:28.104023 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094361    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 05:02:28.104023 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094364    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 05:02:28.104023 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094367    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 05:02:28.104023 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094369    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 05:02:28.104023 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094372    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 05:02:28.104023 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094375    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 05:02:28.104023 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094378    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 05:02:28.104023 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094380    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 05:02:28.104023 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094383    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 05:02:28.104023 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094386    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 05:02:28.104023 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094388    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 05:02:28.104023 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094390    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 05:02:28.104023 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094393    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 05:02:28.104023 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094395    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 05:02:28.104023 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094398    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 05:02:28.104023 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094400    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 05:02:28.104023 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094403    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 05:02:28.104023 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094405    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 05:02:28.104518 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094408    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 05:02:28.104518 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094411    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 05:02:28.104518 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094413    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 05:02:28.104518 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094416    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 05:02:28.104518 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094418    2571 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 05:02:28.104518 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094421    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 05:02:28.104518 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094423    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 05:02:28.104518 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094426    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 05:02:28.104518 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094428    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 05:02:28.104518 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094431    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 05:02:28.104518 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094433    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 05:02:28.104518 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094438    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 05:02:28.104518 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094441    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 05:02:28.104518 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094443    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 05:02:28.104518 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094446    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 05:02:28.104518 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094462    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 05:02:28.104518 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094465    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 05:02:28.104518 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094468    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 05:02:28.104518 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094470    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 05:02:28.104518 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094473    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 05:02:28.104966 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094476    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 05:02:28.104966 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094478    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 05:02:28.104966 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094481    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 05:02:28.104966 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094483    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 05:02:28.104966 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094486    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 05:02:28.104966 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094488    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 05:02:28.104966 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094491    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 05:02:28.104966 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094493    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 05:02:28.104966 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094495    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 05:02:28.104966 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094498    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 05:02:28.104966 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094502    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 05:02:28.104966 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094506    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 05:02:28.104966 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094510    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 05:02:28.104966 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094514    2571 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 05:02:28.104966 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094517    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 05:02:28.104966 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094519    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 05:02:28.104966 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094522    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 05:02:28.104966 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094524    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 05:02:28.104966 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094526    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 05:02:28.104966 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094529    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 05:02:28.105471 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094531    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 05:02:28.105471 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094534    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 05:02:28.105471 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094536    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 05:02:28.105471 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094540    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 05:02:28.105471 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094543    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 05:02:28.105471 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094546    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 05:02:28.105471 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094548    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 05:02:28.105471 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094550    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 05:02:28.105471 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094553    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 05:02:28.105471 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094555    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 05:02:28.105471 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094558    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 05:02:28.105471 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094560    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 05:02:28.105471 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094565    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 05:02:28.105471 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094568    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 05:02:28.105471 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094571    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 05:02:28.105471 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094573    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 05:02:28.105471 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094575    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 05:02:28.105471 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094578    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 05:02:28.105471 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094580    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 05:02:28.105925 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094583    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 05:02:28.105925 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094585    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 05:02:28.105925 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094588    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 05:02:28.105925 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094590    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 05:02:28.105925 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094593    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 05:02:28.105925 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094595    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 05:02:28.105925 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094599    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 05:02:28.105925 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094602    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 05:02:28.105925 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094605    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 05:02:28.105925 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094607    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 05:02:28.105925 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094610    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 05:02:28.105925 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094612    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 05:02:28.105925 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094614    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 05:02:28.105925 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094617    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 05:02:28.105925 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094619    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 05:02:28.105925 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094622    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 05:02:28.105925 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094625    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 05:02:28.105925 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094629    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 05:02:28.105925 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094631    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 05:02:28.106367 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:28.094634    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 05:02:28.106367 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.094688    2571 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 01 05:02:28.106367 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.097103    2571 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 01 05:02:28.106367 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.098047    2571 server.go:1014] "Starting client certificate rotation"
Nov 01 05:02:28.106367 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.098147    2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 01 05:02:28.106367 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.098979    2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 01 05:02:28.106367 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.100122    2571 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 05:02:28.106367 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.102931    2571 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 05:02:28.113222 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.113208    2571 log.go:25] "Validated CRI v1 runtime API"
Nov 01 05:02:28.116857 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.116841    2571 log.go:25] "Validated CRI v1 image API"
Nov 01 05:02:28.117720 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.117689    2571 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 01 05:02:28.120253 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.120234    2571 fs.go:135] Filesystem UUIDs: map[31536baf-b501-4d7f-869c-69829feada7c:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 8c206422-f772-4760-9f25-c27d9444a6f6:/dev/nvme0n1p4]
Nov 01 05:02:28.120319 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.120252    2571 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 01 05:02:28.126245 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.126131    2571 manager.go:217] Machine: {Timestamp:2026-11-01 05:02:28.124712202 +0000 UTC m=+0.325609544 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100701 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec226b1316f4d753c69b8802ebfe9879 SystemUUID:ec226b13-16f4-d753-c69b-8802ebfe9879 BootID:73ecc392-1f88-480c-adc1-253394410c4e Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:11:22:0f:a1:7b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:11:22:0f:a1:7b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:b2:f2:70:e2:43:8a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 01 05:02:28.126245 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.126237    2571 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 01 05:02:28.126378 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.126309    2571 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 01 05:02:28.129761 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.129732    2571 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 01 05:02:28.129882 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.129762    2571 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-92.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 01 05:02:28.129930 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.129884    2571 topology_manager.go:143] "Creating topology manager with none policy"
Nov 01 05:02:28.129930 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.129892    2571 container_manager_linux.go:311] "Creating device plugin manager"
Nov 01 05:02:28.129982 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.129938    2571 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 05:02:28.129982 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.129957    2571 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 05:02:28.131214 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.131203    2571 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 01 05:02:28.135754 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.135742    2571 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 01 05:02:28.135868 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.135859    2571 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 01 05:02:28.136784 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.136775    2571 kubelet.go:493] "Attempting to sync node with API server"
Nov 01 05:02:28.136818 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.136788    2571 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 01 05:02:28.136818 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.136799    2571 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 01 05:02:28.136818 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.136807    2571 kubelet.go:405] "Adding apiserver pod source"
Nov 01 05:02:28.136818 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.136815    2571 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 01 05:02:28.137098 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.137085    2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 05:02:28.139412 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.139383    2571 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 01 05:02:28.146549 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.146538    2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 05:02:28.147464 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.147442    2571 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 01 05:02:28.147509 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.147488    2571 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 01 05:02:28.147652 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.147639    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 01 05:02:28.147691 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.147654    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 01 05:02:28.147691 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.147660    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 01 05:02:28.147691 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.147666    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 01 05:02:28.147691 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.147672    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 01 05:02:28.147691 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.147677    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 01 05:02:28.148338 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.148330    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 01 05:02:28.148365 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.148338    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 01 05:02:28.148365 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.148346    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 01 05:02:28.148365 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.148352    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 01 05:02:28.148365 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.148362    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 01 05:02:28.148493 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.148370    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 01 05:02:28.149919 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.149909    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 01 05:02:28.149919 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.149918    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 01 05:02:28.153547 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.153533    2571 server.go:1263] "Started kubelet"
Nov 01 05:02:28.153670 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.153627    2571 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 01 05:02:28.153745 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.153646    2571 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 01 05:02:28.153745 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.153694    2571 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 01 05:02:28.154175 ip-10-0-129-92 systemd[1]: Started Kubernetes Kubelet.
Nov 01 05:02:28.155573 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.155555    2571 server.go:317] "Adding debug handlers to kubelet server"
Nov 01 05:02:28.157668 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.157650    2571 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 01 05:02:28.161264 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:28.161249    2571 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 01 05:02:28.163512 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.163488    2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 01 05:02:28.164176 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.164156    2571 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 01 05:02:28.164307 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:28.164290    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-92.us-west-1.compute.internal\" not found"
Nov 01 05:02:28.164910 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.164896    2571 factory.go:55] Registering systemd factory
Nov 01 05:02:28.164998 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.164915    2571 factory.go:223] Registration of the systemd container factory successfully
Nov 01 05:02:28.164998 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.164947    2571 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 01 05:02:28.165106 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.165095    2571 factory.go:153] Registering CRI-O factory
Nov 01 05:02:28.165106 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.165105    2571 factory.go:223] Registration of the crio container factory successfully
Nov 01 05:02:28.165208 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.165144    2571 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 01 05:02:28.165208 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.165160    2571 factory.go:103] Registering Raw factory
Nov 01 05:02:28.165208 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.165174    2571 manager.go:1199] Started watching for new ooms in manager
Nov 01 05:02:28.165412 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.164944    2571 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 01 05:02:28.165480 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.165417    2571 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 01 05:02:28.165587 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.165570    2571 reconstruct.go:97] "Volume reconstruction finished"
Nov 01 05:02:28.165640 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.165590    2571 reconciler.go:29] "Reconciler: start to sync state"
Nov 01 05:02:28.166159 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.166139    2571 manager.go:319] Starting recovery of all containers
Nov 01 05:02:28.177502 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.177485    2571 manager.go:324] Recovery completed
Nov 01 05:02:28.178780 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:28.178761    2571 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Nov 01 05:02:28.181500 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.181487    2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 05:02:28.183015 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.182996    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-92.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 05:02:28.183088 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.183035    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-92.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 05:02:28.183088 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.183050    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-92.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 05:02:28.183585 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.183570    2571 cpu_manager.go:226] "Starting" policy="none"
Nov 01 05:02:28.183585 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.183583    2571 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 01 05:02:28.183691 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.183577    2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 05:02:28.183691 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.183599    2571 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 01 05:02:28.184208 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.184190    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-92.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 05:02:28.184276 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.184218    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-92.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 05:02:28.184276 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.184228    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-92.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 05:02:28.185738 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.185725    2571 policy_none.go:50] "Start"
Nov 01 05:02:28.185738 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.185739    2571 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 01 05:02:28.185815 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.185753    2571 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 01 05:02:28.187671 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.187660    2571 policy_none.go:44] "Start"
Nov 01 05:02:28.220124 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.219998    2571 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 01 05:02:28.224123 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.224110    2571 manager.go:341] "Starting Device Plugin manager"
Nov 01 05:02:28.224174 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:28.224136    2571 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 01 05:02:28.224174 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.224144    2571 server.go:87] "Starting device plugin registration server"
Nov 01 05:02:28.224360 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.224348    2571 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 01 05:02:28.224407 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.224360    2571 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 01 05:02:28.224503 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.224488    2571 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 01 05:02:28.224575 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.224564    2571 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 01 05:02:28.224647 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.224576    2571 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 01 05:02:28.225032 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:28.225012    2571 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 01 05:02:28.225111 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:28.225058    2571 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-92.us-west-1.compute.internal\" not found"
Nov 01 05:02:28.299608 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:28.299583    2571 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-92.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 01 05:02:28.301541 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.299618    2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-92.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 01 05:02:28.301541 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:28.299588    2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-92.us-west-1.compute.internal.18e3d33fee0842a7  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-92.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-92.us-west-1.compute.internal,},FirstTimestamp:2026-11-01 05:02:28.153508519 +0000 UTC m=+0.354405862,LastTimestamp:2026-11-01 05:02:28.153508519 +0000 UTC m=+0.354405862,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-92.us-west-1.compute.internal,}"
Nov 01 05:02:28.305668 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.305637    2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 01 05:02:28.306960 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.306942    2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 01 05:02:28.306960 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.306961    2571 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 01 05:02:28.307067 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.306979    2571 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 01 05:02:28.307067 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:28.307016    2571 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 01 05:02:28.324443 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.324418    2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 05:02:28.325162 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.325110    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-92.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 05:02:28.325162 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.325137    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-92.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 05:02:28.325162 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.325147    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-92.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 05:02:28.325282 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.325168    2571 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-129-92.us-west-1.compute.internal"
Nov 01 05:02:28.377574 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.377529    2571 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 01 05:02:28.377574 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.377563    2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zr6fx"
Nov 01 05:02:28.407759 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.407741    2571 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-92.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-92.us-west-1.compute.internal"]
Nov 01 05:02:28.407824 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.407792    2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 05:02:28.408566 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.408550    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-92.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 05:02:28.408613 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.408581    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-92.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 05:02:28.408613 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.408596    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-92.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 05:02:28.410761 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.410749    2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 05:02:28.410904 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.410890    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-92.us-west-1.compute.internal"
Nov 01 05:02:28.410964 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.410928    2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 05:02:28.411440 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.411426    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-92.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 05:02:28.411440 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.411436    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-92.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 05:02:28.411553 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.411469    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-92.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 05:02:28.411553 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.411480    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-92.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 05:02:28.411553 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.411469    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-92.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 05:02:28.411647 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.411597    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-92.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 05:02:28.413666 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.413650    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-92.us-west-1.compute.internal"
Nov 01 05:02:28.413756 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.413675    2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 05:02:28.414329 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.414315    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-92.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 05:02:28.414401 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.414335    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-92.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 05:02:28.414401 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.414349    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-92.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 05:02:28.456197 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:28.456177    2571 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-92.us-west-1.compute.internal\" not found" node="ip-10-0-129-92.us-west-1.compute.internal"
Nov 01 05:02:28.458513 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:28.458495    2571 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-92.us-west-1.compute.internal\" not found" node="ip-10-0-129-92.us-west-1.compute.internal"
Nov 01 05:02:28.458739 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.458726    2571 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 05:02:28.467180 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.467155    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/99a2e3eae38c794f2284a61da069e044-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-92.us-west-1.compute.internal\" (UID: \"99a2e3eae38c794f2284a61da069e044\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-92.us-west-1.compute.internal"
Nov 01 05:02:28.467257 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.467185    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0742a37cb8250b02dcdb2d2301dda7a1-config\") pod \"kube-apiserver-proxy-ip-10-0-129-92.us-west-1.compute.internal\" (UID: \"0742a37cb8250b02dcdb2d2301dda7a1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-92.us-west-1.compute.internal"
Nov 01 05:02:28.467257 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.467211    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/99a2e3eae38c794f2284a61da069e044-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-92.us-west-1.compute.internal\" (UID: \"99a2e3eae38c794f2284a61da069e044\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-92.us-west-1.compute.internal"
Nov 01 05:02:28.556733 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.556705    2571 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 05:02:28.557402 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.557385    2571 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 05:02:28.557913 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.557895    2571 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-129-92.us-west-1.compute.internal"
Nov 01 05:02:28.557980 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:28.557920    2571 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-92.us-west-1.compute.internal\": node \"ip-10-0-129-92.us-west-1.compute.internal\" not found"
Nov 01 05:02:28.559751 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.559717    2571 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 05:02:28.564802 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.564786    2571 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-92.us-west-1.compute.internal"
Nov 01 05:02:28.568063 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.568046    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/99a2e3eae38c794f2284a61da069e044-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-92.us-west-1.compute.internal\" (UID: \"99a2e3eae38c794f2284a61da069e044\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-92.us-west-1.compute.internal"
Nov 01 05:02:28.568132 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.568071    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/99a2e3eae38c794f2284a61da069e044-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-92.us-west-1.compute.internal\" (UID: \"99a2e3eae38c794f2284a61da069e044\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-92.us-west-1.compute.internal"
Nov 01 05:02:28.568132 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.568093    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0742a37cb8250b02dcdb2d2301dda7a1-config\") pod \"kube-apiserver-proxy-ip-10-0-129-92.us-west-1.compute.internal\" (UID: \"0742a37cb8250b02dcdb2d2301dda7a1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-92.us-west-1.compute.internal"
Nov 01 05:02:28.568196 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.568147    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0742a37cb8250b02dcdb2d2301dda7a1-config\") pod \"kube-apiserver-proxy-ip-10-0-129-92.us-west-1.compute.internal\" (UID: \"0742a37cb8250b02dcdb2d2301dda7a1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-92.us-west-1.compute.internal"
Nov 01 05:02:28.568196 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.568151    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/99a2e3eae38c794f2284a61da069e044-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-92.us-west-1.compute.internal\" (UID: \"99a2e3eae38c794f2284a61da069e044\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-92.us-west-1.compute.internal"
Nov 01 05:02:28.568196 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.568151    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/99a2e3eae38c794f2284a61da069e044-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-92.us-west-1.compute.internal\" (UID: \"99a2e3eae38c794f2284a61da069e044\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-92.us-west-1.compute.internal"
Nov 01 05:02:28.625301 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.625284    2571 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 05:02:28.625403 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.625389    2571 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-92.us-west-1.compute.internal"
Nov 01 05:02:28.682319 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.682276    2571 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 05:02:28.759004 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.758984    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-92.us-west-1.compute.internal"
Nov 01 05:02:28.761152 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:28.761135    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-92.us-west-1.compute.internal"
Nov 01 05:02:29.098121 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.098046    2571 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 01 05:02:29.098628 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.098151    2571 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 01 05:02:29.098628 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.098211    2571 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 01 05:02:29.098628 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.098232    2571 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 01 05:02:29.098628 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:29.098226    2571 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://aa20f316a090d4ba282b8462800e02e0-246993faf7c49e49.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-129-92.us-west-1.compute.internal.18e3d33fefcb1fa7\": read tcp 10.0.129.92:60460->18.117.66.57:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-129-92.us-west-1.compute.internal.18e3d33fefcb1fa7  default   5218 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-92.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-129-92.us-west-1.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-129-92.us-west-1.compute.internal,},FirstTimestamp:2026-11-01 05:02:28 +0000 UTC,LastTimestamp:2026-11-01 05:02:28.325150333 +0000 UTC m=+0.526047674,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-92.us-west-1.compute.internal,}"
Nov 01 05:02:29.137393 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.137376    2571 apiserver.go:52] "Watching apiserver"
Nov 01 05:02:29.163925 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.163893    2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 01 05:02:29.382402 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.382320    2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 04:57:29 +0000 UTC" deadline="2028-05-23 15:48:25.679242682 +0000 UTC"
Nov 01 05:02:29.382402 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.382347    2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13666h45m56.296898365s"
Nov 01 05:02:29.411276 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.411257    2571 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 01 05:02:29.411564 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.411540    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-2982h","kube-system/konnectivity-agent-hzr4h","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-92.us-west-1.compute.internal","openshift-multus/multus-5cv94","openshift-cluster-node-tuning-operator/tuned-zm57v","openshift-dns/node-resolver-ddc5d","openshift-network-diagnostics/network-check-target-sr4n8","openshift-network-operator/iptables-alerter-d8jfj","kube-system/global-pull-secret-syncer-9pcxf","kube-system/kube-apiserver-proxy-ip-10-0-129-92.us-west-1.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6kb4c","openshift-image-registry/node-ca-vrbks","openshift-multus/network-metrics-daemon-267rv","openshift-multus/multus-additional-cni-plugins-r4nh2"]
Nov 01 05:02:29.414053 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.414039    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:02:29.416161 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.416142    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hzr4h"
Nov 01 05:02:29.418303 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.418288    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5cv94"
Nov 01 05:02:29.420337 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.420322    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:02:29.422391 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.422378    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ddc5d"
Nov 01 05:02:29.424489 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.424442    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sr4n8"
Nov 01 05:02:29.424570 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:29.424553    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sr4n8" podUID="bcc64278-da5e-4525-90bf-abddde870d70"
Nov 01 05:02:29.426663 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.426648    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-d8jfj"
Nov 01 05:02:29.428992 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.428806    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9pcxf"
Nov 01 05:02:29.428992 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:29.428872    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9pcxf" podUID="e856df5a-5c5e-4421-ae7d-80a760bc72e2"
Nov 01 05:02:29.430985 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.430968    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6kb4c"
Nov 01 05:02:29.433124 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.433111    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vrbks"
Nov 01 05:02:29.435307 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.435288    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-267rv"
Nov 01 05:02:29.435379 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:29.435354    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-267rv" podUID="4321e0ae-e339-40f2-875a-941b1183e766"
Nov 01 05:02:29.437438 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.437424    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-r4nh2"
Nov 01 05:02:29.467589 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.467523    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 01 05:02:29.467589 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.467545    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4vjvv\""
Nov 01 05:02:29.467747 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.467598    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 01 05:02:29.467747 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.467613    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 01 05:02:29.468281 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.468259    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 01 05:02:29.468403 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.468337    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 01 05:02:29.468505 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.468428    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 01 05:02:29.469103 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.469083    2571 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 01 05:02:29.469478 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.469444    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 01 05:02:29.470014 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.469973    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-tw68j\""
Nov 01 05:02:29.470014 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.469971    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 01 05:02:29.471114 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.471095    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 01 05:02:29.471246 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.471136    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 01 05:02:29.471246 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.471146    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 01 05:02:29.471324 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.471314    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 01 05:02:29.471380 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.471350    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-nkwq4\""
Nov 01 05:02:29.473714 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.473689    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a9bb6df5-0993-4640-81cc-52426b72fb3e-cnibin\") pod \"multus-additional-cni-plugins-r4nh2\" (UID: \"a9bb6df5-0993-4640-81cc-52426b72fb3e\") " pod="openshift-multus/multus-additional-cni-plugins-r4nh2"
Nov 01 05:02:29.473763 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.473722    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/972b129f-0166-4705-a71c-09ee0e7a91b9-multus-conf-dir\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:02:29.473763 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.473739    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-host-kubelet\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:02:29.473836 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.473758    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-host-run-ovn-kubernetes\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:02:29.473836 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.473781    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-host-cni-netd\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:02:29.473836 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.473826    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmrx6\" (UniqueName: \"kubernetes.io/projected/7e9cccdd-6c18-4a60-837f-4907411a1c44-kube-api-access-tmrx6\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:02:29.473925 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.473857    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4321e0ae-e339-40f2-875a-941b1183e766-metrics-certs\") pod \"network-metrics-daemon-267rv\" (UID: \"4321e0ae-e339-40f2-875a-941b1183e766\") " pod="openshift-multus/network-metrics-daemon-267rv"
Nov 01 05:02:29.473925 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.473873    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6149f269-49c2-4572-8f6c-fe01e9f0dcbf-host-slash\") pod \"iptables-alerter-d8jfj\" (UID: \"6149f269-49c2-4572-8f6c-fe01e9f0dcbf\") " pod="openshift-network-operator/iptables-alerter-d8jfj"
Nov 01 05:02:29.473925 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.473904    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a9bb6df5-0993-4640-81cc-52426b72fb3e-system-cni-dir\") pod \"multus-additional-cni-plugins-r4nh2\" (UID: \"a9bb6df5-0993-4640-81cc-52426b72fb3e\") " pod="openshift-multus/multus-additional-cni-plugins-r4nh2"
Nov 01 05:02:29.474011 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.473926    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9752ded2-725f-4035-b374-602e4b4eb576-sys\") pod \"tuned-zm57v\" (UID: \"9752ded2-725f-4035-b374-602e4b4eb576\") " pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:02:29.474011 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.473945    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffjr4\" (UniqueName: \"kubernetes.io/projected/bcc64278-da5e-4525-90bf-abddde870d70-kube-api-access-ffjr4\") pod \"network-check-target-sr4n8\" (UID: \"bcc64278-da5e-4525-90bf-abddde870d70\") " pod="openshift-network-diagnostics/network-check-target-sr4n8"
Nov 01 05:02:29.474011 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.473960    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e856df5a-5c5e-4421-ae7d-80a760bc72e2-kubelet-config\") pod \"global-pull-secret-syncer-9pcxf\" (UID: \"e856df5a-5c5e-4421-ae7d-80a760bc72e2\") " pod="kube-system/global-pull-secret-syncer-9pcxf"
Nov 01 05:02:29.474011 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.473981    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9752ded2-725f-4035-b374-602e4b4eb576-host\") pod \"tuned-zm57v\" (UID: \"9752ded2-725f-4035-b374-602e4b4eb576\") " pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:02:29.474129 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.474012    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9752ded2-725f-4035-b374-602e4b4eb576-tmp\") pod \"tuned-zm57v\" (UID: \"9752ded2-725f-4035-b374-602e4b4eb576\") " pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:02:29.474129 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.474029    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/972b129f-0166-4705-a71c-09ee0e7a91b9-host-var-lib-kubelet\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:02:29.474129 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.474044    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9752ded2-725f-4035-b374-602e4b4eb576-etc-sysconfig\") pod \"tuned-zm57v\" (UID: \"9752ded2-725f-4035-b374-602e4b4eb576\") " pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:02:29.474129 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.474067    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9752ded2-725f-4035-b374-602e4b4eb576-etc-kubernetes\") pod \"tuned-zm57v\" (UID: \"9752ded2-725f-4035-b374-602e4b4eb576\") " pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:02:29.474129 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.474091    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9752ded2-725f-4035-b374-602e4b4eb576-etc-sysctl-d\") pod \"tuned-zm57v\" (UID: \"9752ded2-725f-4035-b374-602e4b4eb576\") " pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:02:29.474129 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.474108    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-var-lib-openvswitch\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:02:29.474293 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.474135    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-run-openvswitch\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:02:29.474293 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.474153    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9752ded2-725f-4035-b374-602e4b4eb576-var-lib-kubelet\") pod \"tuned-zm57v\" (UID: \"9752ded2-725f-4035-b374-602e4b4eb576\") " pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:02:29.474293 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.474180    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/972b129f-0166-4705-a71c-09ee0e7a91b9-cni-binary-copy\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:02:29.474293 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.474205    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n95qs\" (UniqueName: \"kubernetes.io/projected/a9bb6df5-0993-4640-81cc-52426b72fb3e-kube-api-access-n95qs\") pod \"multus-additional-cni-plugins-r4nh2\" (UID: \"a9bb6df5-0993-4640-81cc-52426b72fb3e\") " pod="openshift-multus/multus-additional-cni-plugins-r4nh2"
Nov 01 05:02:29.474293 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.474223    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0e24f687-655b-4edb-8901-35412d3fc79b-registration-dir\") pod \"aws-ebs-csi-driver-node-6kb4c\" (UID: \"0e24f687-655b-4edb-8901-35412d3fc79b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6kb4c"
Nov 01 05:02:29.474293 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.474237    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d454e879-03a2-4fb9-8223-79a1aa9d85d2-serviceca\") pod \"node-ca-vrbks\" (UID: \"d454e879-03a2-4fb9-8223-79a1aa9d85d2\") " pod="openshift-image-registry/node-ca-vrbks"
Nov 01 05:02:29.474293 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.474252    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-node-log\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:02:29.474521 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.474265    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7e9cccdd-6c18-4a60-837f-4907411a1c44-ovnkube-script-lib\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:02:29.474521 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.474315    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e856df5a-5c5e-4421-ae7d-80a760bc72e2-original-pull-secret\") pod \"global-pull-secret-syncer-9pcxf\" (UID: \"e856df5a-5c5e-4421-ae7d-80a760bc72e2\") " pod="kube-system/global-pull-secret-syncer-9pcxf"
Nov 01 05:02:29.474521 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.474331    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc8bg\" (UniqueName: \"kubernetes.io/projected/a877f895-a4cf-404d-87e4-440d4b73a870-kube-api-access-xc8bg\") pod \"node-resolver-ddc5d\" (UID: \"a877f895-a4cf-404d-87e4-440d4b73a870\") " pod="openshift-dns/node-resolver-ddc5d"
Nov 01 05:02:29.474521 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.474346    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6149f269-49c2-4572-8f6c-fe01e9f0dcbf-iptables-alerter-script\") pod \"iptables-alerter-d8jfj\" (UID: \"6149f269-49c2-4572-8f6c-fe01e9f0dcbf\") " pod="openshift-network-operator/iptables-alerter-d8jfj"
Nov 01 05:02:29.474521 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.474360    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/972b129f-0166-4705-a71c-09ee0e7a91b9-host-run-k8s-cni-cncf-io\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:02:29.474521 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.474395    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/972b129f-0166-4705-a71c-09ee0e7a91b9-etc-kubernetes\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:02:29.474521 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.474414    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9752ded2-725f-4035-b374-602e4b4eb576-etc-systemd\") pod \"tuned-zm57v\" (UID: \"9752ded2-725f-4035-b374-602e4b4eb576\") " pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:02:29.474521 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.474429    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/972b129f-0166-4705-a71c-09ee0e7a91b9-multus-daemon-config\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:02:29.474521 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.474476    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgfc6\" (UniqueName: \"kubernetes.io/projected/0e24f687-655b-4edb-8901-35412d3fc79b-kube-api-access-tgfc6\") pod \"aws-ebs-csi-driver-node-6kb4c\" (UID: \"0e24f687-655b-4edb-8901-35412d3fc79b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6kb4c"
Nov 01 05:02:29.474521 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.474501    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/972b129f-0166-4705-a71c-09ee0e7a91b9-cnibin\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:02:29.474521 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.474516    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a877f895-a4cf-404d-87e4-440d4b73a870-hosts-file\") pod \"node-resolver-ddc5d\" (UID: \"a877f895-a4cf-404d-87e4-440d4b73a870\") " pod="openshift-dns/node-resolver-ddc5d"
Nov 01 05:02:29.474810 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.474546    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/972b129f-0166-4705-a71c-09ee0e7a91b9-host-var-lib-cni-bin\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:02:29.474810 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.474572    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/972b129f-0166-4705-a71c-09ee0e7a91b9-host-run-multus-certs\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:02:29.474810 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.474592    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0e24f687-655b-4edb-8901-35412d3fc79b-socket-dir\") pod \"aws-ebs-csi-driver-node-6kb4c\" (UID: \"0e24f687-655b-4edb-8901-35412d3fc79b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6kb4c"
Nov 01 05:02:29.474810 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.474607    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0e24f687-655b-4edb-8901-35412d3fc79b-sys-fs\") pod \"aws-ebs-csi-driver-node-6kb4c\" (UID: \"0e24f687-655b-4edb-8901-35412d3fc79b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6kb4c"
Nov 01 05:02:29.474810 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.474620    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-log-socket\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:02:29.474810 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.474634    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a9bb6df5-0993-4640-81cc-52426b72fb3e-cni-binary-copy\") pod \"multus-additional-cni-plugins-r4nh2\" (UID: \"a9bb6df5-0993-4640-81cc-52426b72fb3e\") " pod="openshift-multus/multus-additional-cni-plugins-r4nh2"
Nov 01 05:02:29.474810 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.474649    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a9bb6df5-0993-4640-81cc-52426b72fb3e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-r4nh2\" (UID: \"a9bb6df5-0993-4640-81cc-52426b72fb3e\") " pod="openshift-multus/multus-additional-cni-plugins-r4nh2"
Nov 01 05:02:29.474810 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.474664    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/972b129f-0166-4705-a71c-09ee0e7a91b9-multus-cni-dir\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:02:29.474810 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.474678    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/972b129f-0166-4705-a71c-09ee0e7a91b9-host-run-netns\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:02:29.474810 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.474702    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9752ded2-725f-4035-b374-602e4b4eb576-etc-sysctl-conf\") pod \"tuned-zm57v\" (UID: \"9752ded2-725f-4035-b374-602e4b4eb576\") " pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:02:29.474810 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.474725    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a9bb6df5-0993-4640-81cc-52426b72fb3e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-r4nh2\" (UID: \"a9bb6df5-0993-4640-81cc-52426b72fb3e\") " pod="openshift-multus/multus-additional-cni-plugins-r4nh2"
Nov 01 05:02:29.474810 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.474744    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cca47ebc-569a-4558-af41-5e3755605b9d-konnectivity-ca\") pod \"konnectivity-agent-hzr4h\" (UID: \"cca47ebc-569a-4558-af41-5e3755605b9d\") " pod="kube-system/konnectivity-agent-hzr4h"
Nov 01 05:02:29.474810 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.474765    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/972b129f-0166-4705-a71c-09ee0e7a91b9-system-cni-dir\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:02:29.474810 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.474785    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0e24f687-655b-4edb-8901-35412d3fc79b-device-dir\") pod \"aws-ebs-csi-driver-node-6kb4c\" (UID: \"0e24f687-655b-4edb-8901-35412d3fc79b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6kb4c"
Nov 01 05:02:29.474810 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.474804    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d454e879-03a2-4fb9-8223-79a1aa9d85d2-host\") pod \"node-ca-vrbks\" (UID: \"d454e879-03a2-4fb9-8223-79a1aa9d85d2\") " pod="openshift-image-registry/node-ca-vrbks"
Nov 01 05:02:29.475250 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.474818    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-systemd-units\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:02:29.475250 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.474841    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-host-cni-bin\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:02:29.475250 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.474863    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:02:29.475250 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.474892    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a9bb6df5-0993-4640-81cc-52426b72fb3e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-r4nh2\" (UID: \"a9bb6df5-0993-4640-81cc-52426b72fb3e\") " pod="openshift-multus/multus-additional-cni-plugins-r4nh2"
Nov 01 05:02:29.475250 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.474907    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lv27z\" (UniqueName: \"kubernetes.io/projected/972b129f-0166-4705-a71c-09ee0e7a91b9-kube-api-access-lv27z\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:02:29.475250 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.474921    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9752ded2-725f-4035-b374-602e4b4eb576-etc-modprobe-d\") pod \"tuned-zm57v\" (UID: \"9752ded2-725f-4035-b374-602e4b4eb576\") " pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:02:29.475250 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.474951    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0e24f687-655b-4edb-8901-35412d3fc79b-etc-selinux\") pod \"aws-ebs-csi-driver-node-6kb4c\" (UID: \"0e24f687-655b-4edb-8901-35412d3fc79b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6kb4c"
Nov 01 05:02:29.475250 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.474970    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdlnm\" (UniqueName: \"kubernetes.io/projected/d454e879-03a2-4fb9-8223-79a1aa9d85d2-kube-api-access-hdlnm\") pod \"node-ca-vrbks\" (UID: \"d454e879-03a2-4fb9-8223-79a1aa9d85d2\") " pod="openshift-image-registry/node-ca-vrbks"
Nov 01 05:02:29.475250 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.474990    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7e9cccdd-6c18-4a60-837f-4907411a1c44-ovn-node-metrics-cert\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:02:29.475250 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.475011    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cca47ebc-569a-4558-af41-5e3755605b9d-agent-certs\") pod \"konnectivity-agent-hzr4h\" (UID: \"cca47ebc-569a-4558-af41-5e3755605b9d\") " pod="kube-system/konnectivity-agent-hzr4h"
Nov 01 05:02:29.475250 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.475050    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e856df5a-5c5e-4421-ae7d-80a760bc72e2-dbus\") pod \"global-pull-secret-syncer-9pcxf\" (UID: \"e856df5a-5c5e-4421-ae7d-80a760bc72e2\") " pod="kube-system/global-pull-secret-syncer-9pcxf"
Nov 01 05:02:29.475250 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.475084    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/972b129f-0166-4705-a71c-09ee0e7a91b9-hostroot\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:02:29.475250 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.475102    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9752ded2-725f-4035-b374-602e4b4eb576-run\") pod \"tuned-zm57v\" (UID: \"9752ded2-725f-4035-b374-602e4b4eb576\") " pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:02:29.475250 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.475117    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-run-systemd\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:02:29.475250 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.475146    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7e9cccdd-6c18-4a60-837f-4907411a1c44-ovnkube-config\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:02:29.475250 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.475171    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a877f895-a4cf-404d-87e4-440d4b73a870-tmp-dir\") pod \"node-resolver-ddc5d\" (UID: \"a877f895-a4cf-404d-87e4-440d4b73a870\") " pod="openshift-dns/node-resolver-ddc5d"
Nov 01 05:02:29.475690 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.475188    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntxmg\" (UniqueName: \"kubernetes.io/projected/6149f269-49c2-4572-8f6c-fe01e9f0dcbf-kube-api-access-ntxmg\") pod \"iptables-alerter-d8jfj\" (UID: \"6149f269-49c2-4572-8f6c-fe01e9f0dcbf\") " pod="openshift-network-operator/iptables-alerter-d8jfj"
Nov 01 05:02:29.475690 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.475205    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9752ded2-725f-4035-b374-602e4b4eb576-lib-modules\") pod \"tuned-zm57v\" (UID: \"9752ded2-725f-4035-b374-602e4b4eb576\") " pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:02:29.475690 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.475220    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0e24f687-655b-4edb-8901-35412d3fc79b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6kb4c\" (UID: \"0e24f687-655b-4edb-8901-35412d3fc79b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6kb4c"
Nov 01 05:02:29.475690 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.475244    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-host-run-netns\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:02:29.475690 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.475258    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-etc-openvswitch\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:02:29.475690 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.475275    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-run-ovn\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:02:29.475690 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.475288    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/972b129f-0166-4705-a71c-09ee0e7a91b9-multus-socket-dir-parent\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:02:29.475690 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.475302    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/972b129f-0166-4705-a71c-09ee0e7a91b9-host-var-lib-cni-multus\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:02:29.475690 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.475321    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9752ded2-725f-4035-b374-602e4b4eb576-etc-tuned\") pod \"tuned-zm57v\" (UID: \"9752ded2-725f-4035-b374-602e4b4eb576\") " pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:02:29.475690 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.475340    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rrgq\" (UniqueName: \"kubernetes.io/projected/9752ded2-725f-4035-b374-602e4b4eb576-kube-api-access-8rrgq\") pod \"tuned-zm57v\" (UID: \"9752ded2-725f-4035-b374-602e4b4eb576\") " pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:02:29.475690 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.475362    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-host-slash\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:02:29.475690 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.475376    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7e9cccdd-6c18-4a60-837f-4907411a1c44-env-overrides\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:02:29.475690 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.475392    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wlnj\" (UniqueName: \"kubernetes.io/projected/4321e0ae-e339-40f2-875a-941b1183e766-kube-api-access-2wlnj\") pod \"network-metrics-daemon-267rv\" (UID: \"4321e0ae-e339-40f2-875a-941b1183e766\") " pod="openshift-multus/network-metrics-daemon-267rv"
Nov 01 05:02:29.515172 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.515149    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-q6tm4\""
Nov 01 05:02:29.565083 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.565065    2571 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 01 05:02:29.565159 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.565116    2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-vwrgz"
Nov 01 05:02:29.567948 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.567929    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6ndpw\""
Nov 01 05:02:29.568029 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.567948    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 01 05:02:29.568029 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.567968    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 01 05:02:29.568029 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.567971    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-b9csn\""
Nov 01 05:02:29.568162 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.568050    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 01 05:02:29.568162 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.568086    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 01 05:02:29.568162 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.568100    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 01 05:02:29.568162 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.568159    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-fhbr8\""
Nov 01 05:02:29.568339 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.568167    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 01 05:02:29.568339 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.568192    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-gnn2d\""
Nov 01 05:02:29.568339 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.568195    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 01 05:02:29.568339 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.568171    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 01 05:02:29.568339 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.568216    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 01 05:02:29.568339 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.568243    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 01 05:02:29.568339 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.568269    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 01 05:02:29.568339 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.568272    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-fbhl7\""
Nov 01 05:02:29.568339 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.568283    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 01 05:02:29.568339 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.568303    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 01 05:02:29.568339 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.568280    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 01 05:02:29.568339 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.568314    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 01 05:02:29.575547 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.575532    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/972b129f-0166-4705-a71c-09ee0e7a91b9-hostroot\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:02:29.575609 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.575560    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9752ded2-725f-4035-b374-602e4b4eb576-run\") pod \"tuned-zm57v\" (UID: \"9752ded2-725f-4035-b374-602e4b4eb576\") " pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:02:29.575609 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.575575    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-run-systemd\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:02:29.575609 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.575589    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7e9cccdd-6c18-4a60-837f-4907411a1c44-ovnkube-config\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:02:29.575609 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.575603    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a877f895-a4cf-404d-87e4-440d4b73a870-tmp-dir\") pod \"node-resolver-ddc5d\" (UID: \"a877f895-a4cf-404d-87e4-440d4b73a870\") " pod="openshift-dns/node-resolver-ddc5d"
Nov 01 05:02:29.575796 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.575617    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntxmg\" (UniqueName: \"kubernetes.io/projected/6149f269-49c2-4572-8f6c-fe01e9f0dcbf-kube-api-access-ntxmg\") pod \"iptables-alerter-d8jfj\" (UID: \"6149f269-49c2-4572-8f6c-fe01e9f0dcbf\") " pod="openshift-network-operator/iptables-alerter-d8jfj"
Nov 01 05:02:29.575796 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.575637    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9752ded2-725f-4035-b374-602e4b4eb576-lib-modules\") pod \"tuned-zm57v\" (UID: \"9752ded2-725f-4035-b374-602e4b4eb576\") " pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:02:29.575796 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.575646    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/972b129f-0166-4705-a71c-09ee0e7a91b9-hostroot\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:02:29.575796 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.575658    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0e24f687-655b-4edb-8901-35412d3fc79b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6kb4c\" (UID: \"0e24f687-655b-4edb-8901-35412d3fc79b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6kb4c"
Nov 01 05:02:29.575796 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.575690    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0e24f687-655b-4edb-8901-35412d3fc79b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6kb4c\" (UID: \"0e24f687-655b-4edb-8901-35412d3fc79b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6kb4c"
Nov 01 05:02:29.575796 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.575705    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9752ded2-725f-4035-b374-602e4b4eb576-run\") pod \"tuned-zm57v\" (UID: \"9752ded2-725f-4035-b374-602e4b4eb576\") " pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:02:29.575796 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.575718    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-host-run-netns\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:02:29.575796 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.575659    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-run-systemd\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:02:29.575796 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.575741    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-host-run-netns\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:02:29.575796 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.575745    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-etc-openvswitch\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:02:29.575796 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.575773    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-etc-openvswitch\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:02:29.575796 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.575777    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-run-ovn\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:02:29.575796 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.575783    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9752ded2-725f-4035-b374-602e4b4eb576-lib-modules\") pod \"tuned-zm57v\" (UID: \"9752ded2-725f-4035-b374-602e4b4eb576\") " pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:02:29.576278 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.575800    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/972b129f-0166-4705-a71c-09ee0e7a91b9-multus-socket-dir-parent\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:02:29.576278 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.575824    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-run-ovn\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:02:29.576278 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.575828    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/972b129f-0166-4705-a71c-09ee0e7a91b9-host-var-lib-cni-multus\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:02:29.576278 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.575858    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9752ded2-725f-4035-b374-602e4b4eb576-etc-tuned\") pod \"tuned-zm57v\" (UID: \"9752ded2-725f-4035-b374-602e4b4eb576\") " pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:02:29.576278 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.575898    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a877f895-a4cf-404d-87e4-440d4b73a870-tmp-dir\") pod \"node-resolver-ddc5d\" (UID: \"a877f895-a4cf-404d-87e4-440d4b73a870\") " pod="openshift-dns/node-resolver-ddc5d"
Nov 01 05:02:29.576278 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.575897    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rrgq\" (UniqueName: \"kubernetes.io/projected/9752ded2-725f-4035-b374-602e4b4eb576-kube-api-access-8rrgq\") pod \"tuned-zm57v\" (UID: \"9752ded2-725f-4035-b374-602e4b4eb576\") " pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:02:29.576278 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.575901    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/972b129f-0166-4705-a71c-09ee0e7a91b9-host-var-lib-cni-multus\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:02:29.576278 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.575987    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-host-slash\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:02:29.576278 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.575835    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/972b129f-0166-4705-a71c-09ee0e7a91b9-multus-socket-dir-parent\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:02:29.576278 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.576017    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7e9cccdd-6c18-4a60-837f-4907411a1c44-env-overrides\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:02:29.576278 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.576010    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-host-slash\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:02:29.576278 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.576042    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wlnj\" (UniqueName: \"kubernetes.io/projected/4321e0ae-e339-40f2-875a-941b1183e766-kube-api-access-2wlnj\") pod \"network-metrics-daemon-267rv\" (UID: \"4321e0ae-e339-40f2-875a-941b1183e766\") " pod="openshift-multus/network-metrics-daemon-267rv"
Nov 01 05:02:29.576278 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.576066    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a9bb6df5-0993-4640-81cc-52426b72fb3e-cnibin\") pod \"multus-additional-cni-plugins-r4nh2\" (UID: \"a9bb6df5-0993-4640-81cc-52426b72fb3e\") " pod="openshift-multus/multus-additional-cni-plugins-r4nh2"
Nov 01 05:02:29.576278 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.576090    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/972b129f-0166-4705-a71c-09ee0e7a91b9-multus-conf-dir\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:02:29.576278 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.576138    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a9bb6df5-0993-4640-81cc-52426b72fb3e-cnibin\") pod \"multus-additional-cni-plugins-r4nh2\" (UID: \"a9bb6df5-0993-4640-81cc-52426b72fb3e\") " pod="openshift-multus/multus-additional-cni-plugins-r4nh2"
Nov 01 05:02:29.576278 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.576142    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/972b129f-0166-4705-a71c-09ee0e7a91b9-multus-conf-dir\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:02:29.576278 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.576161    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-host-kubelet\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:02:29.576278 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.576154    2571 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 01 05:02:29.577047 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.576192    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-host-run-ovn-kubernetes\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:02:29.577047 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.576213    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-host-cni-netd\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:02:29.577047 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.576228    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmrx6\" (UniqueName: \"kubernetes.io/projected/7e9cccdd-6c18-4a60-837f-4907411a1c44-kube-api-access-tmrx6\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:02:29.577047 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.576243    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4321e0ae-e339-40f2-875a-941b1183e766-metrics-certs\") pod \"network-metrics-daemon-267rv\" (UID: \"4321e0ae-e339-40f2-875a-941b1183e766\") " pod="openshift-multus/network-metrics-daemon-267rv"
Nov 01 05:02:29.577047 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.576181    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7e9cccdd-6c18-4a60-837f-4907411a1c44-ovnkube-config\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:02:29.577047 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.576264    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6149f269-49c2-4572-8f6c-fe01e9f0dcbf-host-slash\") pod \"iptables-alerter-d8jfj\" (UID: \"6149f269-49c2-4572-8f6c-fe01e9f0dcbf\") " pod="openshift-network-operator/iptables-alerter-d8jfj"
Nov 01 05:02:29.577047 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.576289    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a9bb6df5-0993-4640-81cc-52426b72fb3e-system-cni-dir\") pod \"multus-additional-cni-plugins-r4nh2\" (UID: \"a9bb6df5-0993-4640-81cc-52426b72fb3e\") " pod="openshift-multus/multus-additional-cni-plugins-r4nh2"
Nov 01 05:02:29.577047 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.576290    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-host-cni-netd\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:02:29.577047 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.576294    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-host-run-ovn-kubernetes\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:02:29.577047 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.576180    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-host-kubelet\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:02:29.577047 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.576335    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6149f269-49c2-4572-8f6c-fe01e9f0dcbf-host-slash\") pod \"iptables-alerter-d8jfj\" (UID: \"6149f269-49c2-4572-8f6c-fe01e9f0dcbf\") " pod="openshift-network-operator/iptables-alerter-d8jfj"
Nov 01 05:02:29.577047 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.576335    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a9bb6df5-0993-4640-81cc-52426b72fb3e-system-cni-dir\") pod \"multus-additional-cni-plugins-r4nh2\" (UID: \"a9bb6df5-0993-4640-81cc-52426b72fb3e\") " pod="openshift-multus/multus-additional-cni-plugins-r4nh2"
Nov 01 05:02:29.577047 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:29.576396    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 05:02:29.577047 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.576404    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7e9cccdd-6c18-4a60-837f-4907411a1c44-env-overrides\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:02:29.577047 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.576419    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9752ded2-725f-4035-b374-602e4b4eb576-sys\") pod \"tuned-zm57v\" (UID: \"9752ded2-725f-4035-b374-602e4b4eb576\") " pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:02:29.577047 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.576467    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9752ded2-725f-4035-b374-602e4b4eb576-sys\") pod \"tuned-zm57v\" (UID: \"9752ded2-725f-4035-b374-602e4b4eb576\") " pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:02:29.577047 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.576472    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffjr4\" (UniqueName: \"kubernetes.io/projected/bcc64278-da5e-4525-90bf-abddde870d70-kube-api-access-ffjr4\") pod \"network-check-target-sr4n8\" (UID: \"bcc64278-da5e-4525-90bf-abddde870d70\") " pod="openshift-network-diagnostics/network-check-target-sr4n8"
Nov 01 05:02:29.577047 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.576499    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e856df5a-5c5e-4421-ae7d-80a760bc72e2-kubelet-config\") pod \"global-pull-secret-syncer-9pcxf\" (UID: \"e856df5a-5c5e-4421-ae7d-80a760bc72e2\") " pod="kube-system/global-pull-secret-syncer-9pcxf"
Nov 01 05:02:29.577732 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.576515    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9752ded2-725f-4035-b374-602e4b4eb576-host\") pod \"tuned-zm57v\" (UID: \"9752ded2-725f-4035-b374-602e4b4eb576\") " pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:02:29.577732 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.576528    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9752ded2-725f-4035-b374-602e4b4eb576-tmp\") pod \"tuned-zm57v\" (UID: \"9752ded2-725f-4035-b374-602e4b4eb576\") " pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:02:29.577732 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.576550    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/972b129f-0166-4705-a71c-09ee0e7a91b9-host-var-lib-kubelet\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:02:29.577732 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.576572    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9752ded2-725f-4035-b374-602e4b4eb576-etc-sysconfig\") pod \"tuned-zm57v\" (UID: \"9752ded2-725f-4035-b374-602e4b4eb576\") " pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:02:29.577732 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.576572    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e856df5a-5c5e-4421-ae7d-80a760bc72e2-kubelet-config\") pod \"global-pull-secret-syncer-9pcxf\" (UID: \"e856df5a-5c5e-4421-ae7d-80a760bc72e2\") " pod="kube-system/global-pull-secret-syncer-9pcxf"
Nov 01 05:02:29.577732 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.576588    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9752ded2-725f-4035-b374-602e4b4eb576-host\") pod \"tuned-zm57v\" (UID: \"9752ded2-725f-4035-b374-602e4b4eb576\") " pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:02:29.577732 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:29.576616    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4321e0ae-e339-40f2-875a-941b1183e766-metrics-certs podName:4321e0ae-e339-40f2-875a-941b1183e766 nodeName:}" failed. No retries permitted until 2026-11-01 05:02:30.076580577 +0000 UTC m=+2.277477906 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4321e0ae-e339-40f2-875a-941b1183e766-metrics-certs") pod "network-metrics-daemon-267rv" (UID: "4321e0ae-e339-40f2-875a-941b1183e766") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 05:02:29.577732 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.576622    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/972b129f-0166-4705-a71c-09ee0e7a91b9-host-var-lib-kubelet\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:02:29.577732 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.576642    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9752ded2-725f-4035-b374-602e4b4eb576-etc-kubernetes\") pod \"tuned-zm57v\" (UID: \"9752ded2-725f-4035-b374-602e4b4eb576\") " pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:02:29.577732 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.576650    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9752ded2-725f-4035-b374-602e4b4eb576-etc-sysconfig\") pod \"tuned-zm57v\" (UID: \"9752ded2-725f-4035-b374-602e4b4eb576\") " pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:02:29.577732 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.576664    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9752ded2-725f-4035-b374-602e4b4eb576-etc-sysctl-d\") pod \"tuned-zm57v\" (UID: \"9752ded2-725f-4035-b374-602e4b4eb576\") " pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:02:29.577732 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.576679    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9752ded2-725f-4035-b374-602e4b4eb576-etc-kubernetes\") pod \"tuned-zm57v\" (UID: \"9752ded2-725f-4035-b374-602e4b4eb576\") " pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:02:29.577732 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.576685    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-var-lib-openvswitch\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:02:29.577732 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.576707    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-run-openvswitch\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:02:29.577732 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.576728    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9752ded2-725f-4035-b374-602e4b4eb576-var-lib-kubelet\") pod \"tuned-zm57v\" (UID: \"9752ded2-725f-4035-b374-602e4b4eb576\") " pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:02:29.577732 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.576744    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9752ded2-725f-4035-b374-602e4b4eb576-etc-sysctl-d\") pod \"tuned-zm57v\" (UID: \"9752ded2-725f-4035-b374-602e4b4eb576\") " pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:02:29.577732 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.576753    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/972b129f-0166-4705-a71c-09ee0e7a91b9-cni-binary-copy\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:02:29.578496 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.576764    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-run-openvswitch\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:02:29.578496 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.576780    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9752ded2-725f-4035-b374-602e4b4eb576-var-lib-kubelet\") pod \"tuned-zm57v\" (UID: \"9752ded2-725f-4035-b374-602e4b4eb576\") " pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:02:29.578496 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.576787    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n95qs\" (UniqueName: \"kubernetes.io/projected/a9bb6df5-0993-4640-81cc-52426b72fb3e-kube-api-access-n95qs\") pod \"multus-additional-cni-plugins-r4nh2\" (UID: \"a9bb6df5-0993-4640-81cc-52426b72fb3e\") " pod="openshift-multus/multus-additional-cni-plugins-r4nh2"
Nov 01 05:02:29.578496 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.576815    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0e24f687-655b-4edb-8901-35412d3fc79b-registration-dir\") pod \"aws-ebs-csi-driver-node-6kb4c\" (UID: \"0e24f687-655b-4edb-8901-35412d3fc79b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6kb4c"
Nov 01 05:02:29.578496 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.576841    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d454e879-03a2-4fb9-8223-79a1aa9d85d2-serviceca\") pod \"node-ca-vrbks\" (UID: \"d454e879-03a2-4fb9-8223-79a1aa9d85d2\") " pod="openshift-image-registry/node-ca-vrbks"
Nov 01 05:02:29.578496 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.576865    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-node-log\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:02:29.578496 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.576894    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7e9cccdd-6c18-4a60-837f-4907411a1c44-ovnkube-script-lib\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:02:29.578496 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.576925    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e856df5a-5c5e-4421-ae7d-80a760bc72e2-original-pull-secret\") pod \"global-pull-secret-syncer-9pcxf\" (UID: \"e856df5a-5c5e-4421-ae7d-80a760bc72e2\") " pod="kube-system/global-pull-secret-syncer-9pcxf"
Nov 01 05:02:29.578496 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.576966    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc8bg\" (UniqueName: \"kubernetes.io/projected/a877f895-a4cf-404d-87e4-440d4b73a870-kube-api-access-xc8bg\") pod \"node-resolver-ddc5d\" (UID: \"a877f895-a4cf-404d-87e4-440d4b73a870\") " pod="openshift-dns/node-resolver-ddc5d"
Nov 01 05:02:29.578496 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.577000    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6149f269-49c2-4572-8f6c-fe01e9f0dcbf-iptables-alerter-script\") pod \"iptables-alerter-d8jfj\" (UID: \"6149f269-49c2-4572-8f6c-fe01e9f0dcbf\") " pod="openshift-network-operator/iptables-alerter-d8jfj"
Nov 01 05:02:29.578496 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.577034    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/972b129f-0166-4705-a71c-09ee0e7a91b9-host-run-k8s-cni-cncf-io\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:02:29.578496 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.577061    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/972b129f-0166-4705-a71c-09ee0e7a91b9-etc-kubernetes\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:02:29.578496 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.577087    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9752ded2-725f-4035-b374-602e4b4eb576-etc-systemd\") pod \"tuned-zm57v\" (UID: \"9752ded2-725f-4035-b374-602e4b4eb576\") " pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:02:29.578496 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.577115    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/972b129f-0166-4705-a71c-09ee0e7a91b9-multus-daemon-config\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:02:29.578496 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.577143    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgfc6\" (UniqueName: \"kubernetes.io/projected/0e24f687-655b-4edb-8901-35412d3fc79b-kube-api-access-tgfc6\") pod \"aws-ebs-csi-driver-node-6kb4c\" (UID: \"0e24f687-655b-4edb-8901-35412d3fc79b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6kb4c"
Nov 01 05:02:29.578496 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.576720    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-var-lib-openvswitch\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:02:29.578496 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.577167    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/972b129f-0166-4705-a71c-09ee0e7a91b9-cnibin\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:02:29.579785 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.577190    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a877f895-a4cf-404d-87e4-440d4b73a870-hosts-file\") pod \"node-resolver-ddc5d\" (UID: \"a877f895-a4cf-404d-87e4-440d4b73a870\") " pod="openshift-dns/node-resolver-ddc5d"
Nov 01 05:02:29.579785 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.577211    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/972b129f-0166-4705-a71c-09ee0e7a91b9-host-var-lib-cni-bin\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:02:29.579785 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.577217    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/972b129f-0166-4705-a71c-09ee0e7a91b9-cni-binary-copy\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:02:29.579785 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.577236    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/972b129f-0166-4705-a71c-09ee0e7a91b9-host-run-multus-certs\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:02:29.579785 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.577259    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0e24f687-655b-4edb-8901-35412d3fc79b-socket-dir\") pod \"aws-ebs-csi-driver-node-6kb4c\" (UID: \"0e24f687-655b-4edb-8901-35412d3fc79b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6kb4c"
Nov 01 05:02:29.579785 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.577266    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d454e879-03a2-4fb9-8223-79a1aa9d85d2-serviceca\") pod \"node-ca-vrbks\" (UID: \"d454e879-03a2-4fb9-8223-79a1aa9d85d2\") " pod="openshift-image-registry/node-ca-vrbks"
Nov 01 05:02:29.579785 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.577281    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0e24f687-655b-4edb-8901-35412d3fc79b-sys-fs\") pod \"aws-ebs-csi-driver-node-6kb4c\" (UID: \"0e24f687-655b-4edb-8901-35412d3fc79b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6kb4c"
Nov 01 05:02:29.579785 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.577343    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0e24f687-655b-4edb-8901-35412d3fc79b-sys-fs\") pod \"aws-ebs-csi-driver-node-6kb4c\" (UID: \"0e24f687-655b-4edb-8901-35412d3fc79b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6kb4c"
Nov 01 05:02:29.579785 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.577391    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0e24f687-655b-4edb-8901-35412d3fc79b-registration-dir\") pod \"aws-ebs-csi-driver-node-6kb4c\" (UID: \"0e24f687-655b-4edb-8901-35412d3fc79b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6kb4c"
Nov 01 05:02:29.579785 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.577525    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/972b129f-0166-4705-a71c-09ee0e7a91b9-host-var-lib-cni-bin\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:02:29.579785 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.577545    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-log-socket\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:02:29.579785 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.577573    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a9bb6df5-0993-4640-81cc-52426b72fb3e-cni-binary-copy\") pod \"multus-additional-cni-plugins-r4nh2\" (UID: \"a9bb6df5-0993-4640-81cc-52426b72fb3e\") " pod="openshift-multus/multus-additional-cni-plugins-r4nh2"
Nov 01 05:02:29.579785 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.577603    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/972b129f-0166-4705-a71c-09ee0e7a91b9-cnibin\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:02:29.579785 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.577628    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a9bb6df5-0993-4640-81cc-52426b72fb3e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-r4nh2\" (UID: \"a9bb6df5-0993-4640-81cc-52426b72fb3e\") " pod="openshift-multus/multus-additional-cni-plugins-r4nh2"
Nov 01 05:02:29.579785 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.577937    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-node-log\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:02:29.579785 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.578007    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9752ded2-725f-4035-b374-602e4b4eb576-etc-systemd\") pod \"tuned-zm57v\" (UID: \"9752ded2-725f-4035-b374-602e4b4eb576\") " pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:02:29.579785 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.578009    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6149f269-49c2-4572-8f6c-fe01e9f0dcbf-iptables-alerter-script\") pod \"iptables-alerter-d8jfj\" (UID: \"6149f269-49c2-4572-8f6c-fe01e9f0dcbf\") " pod="openshift-network-operator/iptables-alerter-d8jfj"
Nov 01 05:02:29.580668 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.578060    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/972b129f-0166-4705-a71c-09ee0e7a91b9-multus-cni-dir\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:02:29.580668 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.578067    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a877f895-a4cf-404d-87e4-440d4b73a870-hosts-file\") pod \"node-resolver-ddc5d\" (UID: \"a877f895-a4cf-404d-87e4-440d4b73a870\") " pod="openshift-dns/node-resolver-ddc5d"
Nov 01 05:02:29.580668 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.578109    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-log-socket\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:02:29.580668 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.578504    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/972b129f-0166-4705-a71c-09ee0e7a91b9-multus-daemon-config\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:02:29.580668 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.578650    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0e24f687-655b-4edb-8901-35412d3fc79b-socket-dir\") pod \"aws-ebs-csi-driver-node-6kb4c\" (UID: \"0e24f687-655b-4edb-8901-35412d3fc79b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6kb4c"
Nov 01 05:02:29.580668 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.578692    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/972b129f-0166-4705-a71c-09ee0e7a91b9-etc-kubernetes\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:02:29.580668 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.577942    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/972b129f-0166-4705-a71c-09ee0e7a91b9-multus-cni-dir\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:02:29.580668 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.578763    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/972b129f-0166-4705-a71c-09ee0e7a91b9-host-run-k8s-cni-cncf-io\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:02:29.580668 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.578777    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/972b129f-0166-4705-a71c-09ee0e7a91b9-host-run-netns\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:02:29.580668 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.578730    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a9bb6df5-0993-4640-81cc-52426b72fb3e-cni-binary-copy\") pod \"multus-additional-cni-plugins-r4nh2\" (UID: \"a9bb6df5-0993-4640-81cc-52426b72fb3e\") " pod="openshift-multus/multus-additional-cni-plugins-r4nh2"
Nov 01 05:02:29.580668 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:29.578890    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 05:02:29.580668 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:29.578978    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e856df5a-5c5e-4421-ae7d-80a760bc72e2-original-pull-secret podName:e856df5a-5c5e-4421-ae7d-80a760bc72e2 nodeName:}" failed. No retries permitted until 2026-11-01 05:02:30.07896036 +0000 UTC m=+2.279857702 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e856df5a-5c5e-4421-ae7d-80a760bc72e2-original-pull-secret") pod "global-pull-secret-syncer-9pcxf" (UID: "e856df5a-5c5e-4421-ae7d-80a760bc72e2") : object "kube-system"/"original-pull-secret" not registered
Nov 01 05:02:29.580668 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.579201    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a9bb6df5-0993-4640-81cc-52426b72fb3e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-r4nh2\" (UID: \"a9bb6df5-0993-4640-81cc-52426b72fb3e\") " pod="openshift-multus/multus-additional-cni-plugins-r4nh2"
Nov 01 05:02:29.580668 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.579291    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9752ded2-725f-4035-b374-602e4b4eb576-etc-sysctl-conf\") pod \"tuned-zm57v\" (UID: \"9752ded2-725f-4035-b374-602e4b4eb576\") " pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:02:29.580668 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.579317    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/972b129f-0166-4705-a71c-09ee0e7a91b9-host-run-multus-certs\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:02:29.580668 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.579322    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a9bb6df5-0993-4640-81cc-52426b72fb3e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-r4nh2\" (UID: \"a9bb6df5-0993-4640-81cc-52426b72fb3e\") " pod="openshift-multus/multus-additional-cni-plugins-r4nh2"
Nov 01 05:02:29.580668 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.579399    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cca47ebc-569a-4558-af41-5e3755605b9d-konnectivity-ca\") pod \"konnectivity-agent-hzr4h\" (UID: \"cca47ebc-569a-4558-af41-5e3755605b9d\") " pod="kube-system/konnectivity-agent-hzr4h"
Nov 01 05:02:29.581361 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.579473    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/972b129f-0166-4705-a71c-09ee0e7a91b9-system-cni-dir\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:02:29.581361 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.579538    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/972b129f-0166-4705-a71c-09ee0e7a91b9-host-run-netns\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:02:29.581361 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.579654    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9752ded2-725f-4035-b374-602e4b4eb576-etc-sysctl-conf\") pod \"tuned-zm57v\" (UID: \"9752ded2-725f-4035-b374-602e4b4eb576\") " pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:02:29.581361 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.580007    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0e24f687-655b-4edb-8901-35412d3fc79b-device-dir\") pod \"aws-ebs-csi-driver-node-6kb4c\" (UID: \"0e24f687-655b-4edb-8901-35412d3fc79b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6kb4c"
Nov 01 05:02:29.581361 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.580051    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d454e879-03a2-4fb9-8223-79a1aa9d85d2-host\") pod \"node-ca-vrbks\" (UID: \"d454e879-03a2-4fb9-8223-79a1aa9d85d2\") " pod="openshift-image-registry/node-ca-vrbks"
Nov 01 05:02:29.581361 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.580059    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9752ded2-725f-4035-b374-602e4b4eb576-etc-tuned\") pod \"tuned-zm57v\" (UID: \"9752ded2-725f-4035-b374-602e4b4eb576\") " pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:02:29.581361 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.580073    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-systemd-units\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:02:29.581361 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.580096    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-host-cni-bin\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:02:29.581361 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.580104    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a9bb6df5-0993-4640-81cc-52426b72fb3e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-r4nh2\" (UID: \"a9bb6df5-0993-4640-81cc-52426b72fb3e\") " pod="openshift-multus/multus-additional-cni-plugins-r4nh2"
Nov 01 05:02:29.581361 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.580140    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-systemd-units\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:02:29.581361 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.580146    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0e24f687-655b-4edb-8901-35412d3fc79b-device-dir\") pod \"aws-ebs-csi-driver-node-6kb4c\" (UID: \"0e24f687-655b-4edb-8901-35412d3fc79b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6kb4c"
Nov 01 05:02:29.581361 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.580165    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:02:29.581361 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.580172    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/972b129f-0166-4705-a71c-09ee0e7a91b9-system-cni-dir\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:02:29.581361 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.580188    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a9bb6df5-0993-4640-81cc-52426b72fb3e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-r4nh2\" (UID: \"a9bb6df5-0993-4640-81cc-52426b72fb3e\") " pod="openshift-multus/multus-additional-cni-plugins-r4nh2"
Nov 01 05:02:29.581361 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.580204    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv27z\" (UniqueName: \"kubernetes.io/projected/972b129f-0166-4705-a71c-09ee0e7a91b9-kube-api-access-lv27z\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:02:29.581361 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.580212    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:02:29.581361 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.580222    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9752ded2-725f-4035-b374-602e4b4eb576-etc-modprobe-d\") pod \"tuned-zm57v\" (UID: \"9752ded2-725f-4035-b374-602e4b4eb576\") " pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:02:29.582093 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.580226    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-host-cni-bin\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:02:29.582093 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.580260    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0e24f687-655b-4edb-8901-35412d3fc79b-etc-selinux\") pod \"aws-ebs-csi-driver-node-6kb4c\" (UID: \"0e24f687-655b-4edb-8901-35412d3fc79b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6kb4c"
Nov 01 05:02:29.582093 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.580273    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9752ded2-725f-4035-b374-602e4b4eb576-etc-modprobe-d\") pod \"tuned-zm57v\" (UID: \"9752ded2-725f-4035-b374-602e4b4eb576\") " pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:02:29.582093 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.580296    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdlnm\" (UniqueName: \"kubernetes.io/projected/d454e879-03a2-4fb9-8223-79a1aa9d85d2-kube-api-access-hdlnm\") pod \"node-ca-vrbks\" (UID: \"d454e879-03a2-4fb9-8223-79a1aa9d85d2\") " pod="openshift-image-registry/node-ca-vrbks"
Nov 01 05:02:29.582093 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.580328    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7e9cccdd-6c18-4a60-837f-4907411a1c44-ovn-node-metrics-cert\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:02:29.582093 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.580358    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cca47ebc-569a-4558-af41-5e3755605b9d-agent-certs\") pod \"konnectivity-agent-hzr4h\" (UID: \"cca47ebc-569a-4558-af41-5e3755605b9d\") " pod="kube-system/konnectivity-agent-hzr4h"
Nov 01 05:02:29.582093 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.580387    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d454e879-03a2-4fb9-8223-79a1aa9d85d2-host\") pod \"node-ca-vrbks\" (UID: \"d454e879-03a2-4fb9-8223-79a1aa9d85d2\") " pod="openshift-image-registry/node-ca-vrbks"
Nov 01 05:02:29.582093 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.580358    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a9bb6df5-0993-4640-81cc-52426b72fb3e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-r4nh2\" (UID: \"a9bb6df5-0993-4640-81cc-52426b72fb3e\") " pod="openshift-multus/multus-additional-cni-plugins-r4nh2"
Nov 01 05:02:29.582093 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.580385    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e856df5a-5c5e-4421-ae7d-80a760bc72e2-dbus\") pod \"global-pull-secret-syncer-9pcxf\" (UID: \"e856df5a-5c5e-4421-ae7d-80a760bc72e2\") " pod="kube-system/global-pull-secret-syncer-9pcxf"
Nov 01 05:02:29.582093 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.580484    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0e24f687-655b-4edb-8901-35412d3fc79b-etc-selinux\") pod \"aws-ebs-csi-driver-node-6kb4c\" (UID: \"0e24f687-655b-4edb-8901-35412d3fc79b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6kb4c"
Nov 01 05:02:29.582093 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.580518    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e856df5a-5c5e-4421-ae7d-80a760bc72e2-dbus\") pod \"global-pull-secret-syncer-9pcxf\" (UID: \"e856df5a-5c5e-4421-ae7d-80a760bc72e2\") " pod="kube-system/global-pull-secret-syncer-9pcxf"
Nov 01 05:02:29.582093 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.580613    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7e9cccdd-6c18-4a60-837f-4907411a1c44-ovnkube-script-lib\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:02:29.582769 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.582747    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9752ded2-725f-4035-b374-602e4b4eb576-tmp\") pod \"tuned-zm57v\" (UID: \"9752ded2-725f-4035-b374-602e4b4eb576\") " pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:02:29.582905 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.582885    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cca47ebc-569a-4558-af41-5e3755605b9d-konnectivity-ca\") pod \"konnectivity-agent-hzr4h\" (UID: \"cca47ebc-569a-4558-af41-5e3755605b9d\") " pod="kube-system/konnectivity-agent-hzr4h"
Nov 01 05:02:29.584554 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.584532    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7e9cccdd-6c18-4a60-837f-4907411a1c44-ovn-node-metrics-cert\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:02:29.584831 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.584815    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cca47ebc-569a-4558-af41-5e3755605b9d-agent-certs\") pod \"konnectivity-agent-hzr4h\" (UID: \"cca47ebc-569a-4558-af41-5e3755605b9d\") " pod="kube-system/konnectivity-agent-hzr4h"
Nov 01 05:02:29.635244 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:29.635194    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 05:02:29.635244 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:29.635220    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 05:02:29.635244 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:29.635236    2571 projected.go:196] Error preparing data for projected volume kube-api-access-ffjr4 for pod openshift-network-diagnostics/network-check-target-sr4n8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 05:02:29.635394 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:29.635303    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bcc64278-da5e-4525-90bf-abddde870d70-kube-api-access-ffjr4 podName:bcc64278-da5e-4525-90bf-abddde870d70 nodeName:}" failed. No retries permitted until 2026-11-01 05:02:30.135287257 +0000 UTC m=+2.336184624 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-ffjr4" (UniqueName: "kubernetes.io/projected/bcc64278-da5e-4525-90bf-abddde870d70-kube-api-access-ffjr4") pod "network-check-target-sr4n8" (UID: "bcc64278-da5e-4525-90bf-abddde870d70") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 05:02:29.637321 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.637291    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntxmg\" (UniqueName: \"kubernetes.io/projected/6149f269-49c2-4572-8f6c-fe01e9f0dcbf-kube-api-access-ntxmg\") pod \"iptables-alerter-d8jfj\" (UID: \"6149f269-49c2-4572-8f6c-fe01e9f0dcbf\") " pod="openshift-network-operator/iptables-alerter-d8jfj"
Nov 01 05:02:29.637737 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.637694    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rrgq\" (UniqueName: \"kubernetes.io/projected/9752ded2-725f-4035-b374-602e4b4eb576-kube-api-access-8rrgq\") pod \"tuned-zm57v\" (UID: \"9752ded2-725f-4035-b374-602e4b4eb576\") " pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:02:29.637868 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.637848    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n95qs\" (UniqueName: \"kubernetes.io/projected/a9bb6df5-0993-4640-81cc-52426b72fb3e-kube-api-access-n95qs\") pod \"multus-additional-cni-plugins-r4nh2\" (UID: \"a9bb6df5-0993-4640-81cc-52426b72fb3e\") " pod="openshift-multus/multus-additional-cni-plugins-r4nh2"
Nov 01 05:02:29.637958 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.637935    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmrx6\" (UniqueName: \"kubernetes.io/projected/7e9cccdd-6c18-4a60-837f-4907411a1c44-kube-api-access-tmrx6\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:02:29.638226 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.638202    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wlnj\" (UniqueName: \"kubernetes.io/projected/4321e0ae-e339-40f2-875a-941b1183e766-kube-api-access-2wlnj\") pod \"network-metrics-daemon-267rv\" (UID: \"4321e0ae-e339-40f2-875a-941b1183e766\") " pod="openshift-multus/network-metrics-daemon-267rv"
Nov 01 05:02:29.638474 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.638441    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgfc6\" (UniqueName: \"kubernetes.io/projected/0e24f687-655b-4edb-8901-35412d3fc79b-kube-api-access-tgfc6\") pod \"aws-ebs-csi-driver-node-6kb4c\" (UID: \"0e24f687-655b-4edb-8901-35412d3fc79b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6kb4c"
Nov 01 05:02:29.640230 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.640205    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc8bg\" (UniqueName: \"kubernetes.io/projected/a877f895-a4cf-404d-87e4-440d4b73a870-kube-api-access-xc8bg\") pod \"node-resolver-ddc5d\" (UID: \"a877f895-a4cf-404d-87e4-440d4b73a870\") " pod="openshift-dns/node-resolver-ddc5d"
Nov 01 05:02:29.641030 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.641012    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv27z\" (UniqueName: \"kubernetes.io/projected/972b129f-0166-4705-a71c-09ee0e7a91b9-kube-api-access-lv27z\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:02:29.641609 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.641593    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdlnm\" (UniqueName: \"kubernetes.io/projected/d454e879-03a2-4fb9-8223-79a1aa9d85d2-kube-api-access-hdlnm\") pod \"node-ca-vrbks\" (UID: \"d454e879-03a2-4fb9-8223-79a1aa9d85d2\") " pod="openshift-image-registry/node-ca-vrbks"
Nov 01 05:02:29.761507 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.761484    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:02:29.772959 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.772935    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hzr4h"
Nov 01 05:02:29.789652 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.789625    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5cv94"
Nov 01 05:02:29.802228 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.802207    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:02:29.817002 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.816980    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ddc5d"
Nov 01 05:02:29.830839 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.830815    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-d8jfj"
Nov 01 05:02:29.840304 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.840288    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6kb4c"
Nov 01 05:02:29.855983 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.855966    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vrbks"
Nov 01 05:02:29.872264 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.872241    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-r4nh2"
Nov 01 05:02:29.891585 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:29.891522    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0742a37cb8250b02dcdb2d2301dda7a1.slice/crio-35f50f1e7eed82a9c567aa5d24724ef38198719bbf0c8a47d237d2da9df7aaa5 WatchSource:0}: Error finding container 35f50f1e7eed82a9c567aa5d24724ef38198719bbf0c8a47d237d2da9df7aaa5: Status 404 returned error can't find the container with id 35f50f1e7eed82a9c567aa5d24724ef38198719bbf0c8a47d237d2da9df7aaa5
Nov 01 05:02:29.892129 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:29.892107    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99a2e3eae38c794f2284a61da069e044.slice/crio-e96638928e69534905932599d19846f2632b24018a9b807eb5d9c55f0b84bc3a WatchSource:0}: Error finding container e96638928e69534905932599d19846f2632b24018a9b807eb5d9c55f0b84bc3a: Status 404 returned error can't find the container with id e96638928e69534905932599d19846f2632b24018a9b807eb5d9c55f0b84bc3a
Nov 01 05:02:29.892711 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:29.892691    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9bb6df5_0993_4640_81cc_52426b72fb3e.slice/crio-7e06150d8e2d064f862ad246e3da4c8cb964e2306c18fa4b2280dd90c1587af7 WatchSource:0}: Error finding container 7e06150d8e2d064f862ad246e3da4c8cb964e2306c18fa4b2280dd90c1587af7: Status 404 returned error can't find the container with id 7e06150d8e2d064f862ad246e3da4c8cb964e2306c18fa4b2280dd90c1587af7
Nov 01 05:02:29.893320 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:29.893288    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd454e879_03a2_4fb9_8223_79a1aa9d85d2.slice/crio-e7b086d9337668f71e0f1d261a7126a33c053fb5fa12c69ccd7eb3548c960812 WatchSource:0}: Error finding container e7b086d9337668f71e0f1d261a7126a33c053fb5fa12c69ccd7eb3548c960812: Status 404 returned error can't find the container with id e7b086d9337668f71e0f1d261a7126a33c053fb5fa12c69ccd7eb3548c960812
Nov 01 05:02:29.898151 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:29.898069    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 05:02:30.083662 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:30.083638    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4321e0ae-e339-40f2-875a-941b1183e766-metrics-certs\") pod \"network-metrics-daemon-267rv\" (UID: \"4321e0ae-e339-40f2-875a-941b1183e766\") " pod="openshift-multus/network-metrics-daemon-267rv"
Nov 01 05:02:30.083782 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:30.083676    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e856df5a-5c5e-4421-ae7d-80a760bc72e2-original-pull-secret\") pod \"global-pull-secret-syncer-9pcxf\" (UID: \"e856df5a-5c5e-4421-ae7d-80a760bc72e2\") " pod="kube-system/global-pull-secret-syncer-9pcxf"
Nov 01 05:02:30.083782 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:30.083768    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 05:02:30.083851 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:30.083815    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4321e0ae-e339-40f2-875a-941b1183e766-metrics-certs podName:4321e0ae-e339-40f2-875a-941b1183e766 nodeName:}" failed. No retries permitted until 2026-11-01 05:02:31.083801798 +0000 UTC m=+3.284699141 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4321e0ae-e339-40f2-875a-941b1183e766-metrics-certs") pod "network-metrics-daemon-267rv" (UID: "4321e0ae-e339-40f2-875a-941b1183e766") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 05:02:30.083851 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:30.083769    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 05:02:30.083851 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:30.083843    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e856df5a-5c5e-4421-ae7d-80a760bc72e2-original-pull-secret podName:e856df5a-5c5e-4421-ae7d-80a760bc72e2 nodeName:}" failed. No retries permitted until 2026-11-01 05:02:31.083837566 +0000 UTC m=+3.284734895 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e856df5a-5c5e-4421-ae7d-80a760bc72e2-original-pull-secret") pod "global-pull-secret-syncer-9pcxf" (UID: "e856df5a-5c5e-4421-ae7d-80a760bc72e2") : object "kube-system"/"original-pull-secret" not registered
Nov 01 05:02:30.184810 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:30.184746    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffjr4\" (UniqueName: \"kubernetes.io/projected/bcc64278-da5e-4525-90bf-abddde870d70-kube-api-access-ffjr4\") pod \"network-check-target-sr4n8\" (UID: \"bcc64278-da5e-4525-90bf-abddde870d70\") " pod="openshift-network-diagnostics/network-check-target-sr4n8"
Nov 01 05:02:30.185285 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:30.184879    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 05:02:30.185285 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:30.184894    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 05:02:30.185285 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:30.184903    2571 projected.go:196] Error preparing data for projected volume kube-api-access-ffjr4 for pod openshift-network-diagnostics/network-check-target-sr4n8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 05:02:30.185285 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:30.184944    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bcc64278-da5e-4525-90bf-abddde870d70-kube-api-access-ffjr4 podName:bcc64278-da5e-4525-90bf-abddde870d70 nodeName:}" failed. No retries permitted until 2026-11-01 05:02:31.184932546 +0000 UTC m=+3.385829888 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-ffjr4" (UniqueName: "kubernetes.io/projected/bcc64278-da5e-4525-90bf-abddde870d70-kube-api-access-ffjr4") pod "network-check-target-sr4n8" (UID: "bcc64278-da5e-4525-90bf-abddde870d70") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 05:02:30.301216 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:30.301187    2571 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 05:02:30.311448 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:30.311400    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r4nh2" event={"ID":"a9bb6df5-0993-4640-81cc-52426b72fb3e","Type":"ContainerStarted","Data":"7e06150d8e2d064f862ad246e3da4c8cb964e2306c18fa4b2280dd90c1587af7"}
Nov 01 05:02:30.312609 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:30.312588    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vrbks" event={"ID":"d454e879-03a2-4fb9-8223-79a1aa9d85d2","Type":"ContainerStarted","Data":"e7b086d9337668f71e0f1d261a7126a33c053fb5fa12c69ccd7eb3548c960812"}
Nov 01 05:02:30.313591 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:30.313571    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-92.us-west-1.compute.internal" event={"ID":"99a2e3eae38c794f2284a61da069e044","Type":"ContainerStarted","Data":"e96638928e69534905932599d19846f2632b24018a9b807eb5d9c55f0b84bc3a"}
Nov 01 05:02:30.314645 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:30.314626    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-92.us-west-1.compute.internal" event={"ID":"0742a37cb8250b02dcdb2d2301dda7a1","Type":"ContainerStarted","Data":"35f50f1e7eed82a9c567aa5d24724ef38198719bbf0c8a47d237d2da9df7aaa5"}
Nov 01 05:02:30.568642 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:30.568533    2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 04:57:30 +0000 UTC" deadline="2028-05-15 23:39:20.981310541 +0000 UTC"
Nov 01 05:02:30.568642 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:30.568576    2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13482h36m50.412737399s"
Nov 01 05:02:30.581933 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:30.581908    2571 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 05:02:30.646412 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:30.646383    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod972b129f_0166_4705_a71c_09ee0e7a91b9.slice/crio-0a973929be5cebc219b720e4cbc988c2101c4d3885f1252d602d91ece8d10c9b WatchSource:0}: Error finding container 0a973929be5cebc219b720e4cbc988c2101c4d3885f1252d602d91ece8d10c9b: Status 404 returned error can't find the container with id 0a973929be5cebc219b720e4cbc988c2101c4d3885f1252d602d91ece8d10c9b
Nov 01 05:02:30.647034 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:30.646916    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcca47ebc_569a_4558_af41_5e3755605b9d.slice/crio-4eff5ab2517b779c97c79777f957957fe446c9b09aba3853080f6cc1edf21822 WatchSource:0}: Error finding container 4eff5ab2517b779c97c79777f957957fe446c9b09aba3853080f6cc1edf21822: Status 404 returned error can't find the container with id 4eff5ab2517b779c97c79777f957957fe446c9b09aba3853080f6cc1edf21822
Nov 01 05:02:30.648077 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:30.647932    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e9cccdd_6c18_4a60_837f_4907411a1c44.slice/crio-e4e554abfa6a31cdecc3e57264ae7ce20607c5d8b75e65f9b6d2bdd3c3d435e9 WatchSource:0}: Error finding container e4e554abfa6a31cdecc3e57264ae7ce20607c5d8b75e65f9b6d2bdd3c3d435e9: Status 404 returned error can't find the container with id e4e554abfa6a31cdecc3e57264ae7ce20607c5d8b75e65f9b6d2bdd3c3d435e9
Nov 01 05:02:30.678417 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:30.678390    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda877f895_a4cf_404d_87e4_440d4b73a870.slice/crio-fa82b731dbe7fc0fedf4b1e1fc8c3c6dca92546a0a2b97db3d83e7f320bc0bd3 WatchSource:0}: Error finding container fa82b731dbe7fc0fedf4b1e1fc8c3c6dca92546a0a2b97db3d83e7f320bc0bd3: Status 404 returned error can't find the container with id fa82b731dbe7fc0fedf4b1e1fc8c3c6dca92546a0a2b97db3d83e7f320bc0bd3
Nov 01 05:02:30.685257 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:30.685235    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6149f269_49c2_4572_8f6c_fe01e9f0dcbf.slice/crio-5bc3adeb1375285df89458f45d2042ba303bd03e91b726e1d832adcc059030f9 WatchSource:0}: Error finding container 5bc3adeb1375285df89458f45d2042ba303bd03e91b726e1d832adcc059030f9: Status 404 returned error can't find the container with id 5bc3adeb1375285df89458f45d2042ba303bd03e91b726e1d832adcc059030f9
Nov 01 05:02:30.704445 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:30.704423    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9752ded2_725f_4035_b374_602e4b4eb576.slice/crio-c2dfa59500c789ca4744842d55b7f0abfa7266b92f0009d18d6fc18ed125502a WatchSource:0}: Error finding container c2dfa59500c789ca4744842d55b7f0abfa7266b92f0009d18d6fc18ed125502a: Status 404 returned error can't find the container with id c2dfa59500c789ca4744842d55b7f0abfa7266b92f0009d18d6fc18ed125502a
Nov 01 05:02:30.709992 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:30.709966    2571 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 05:02:30.743725 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:02:30.743702    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e24f687_655b_4edb_8901_35412d3fc79b.slice/crio-53da545c04e8752f739145afdc125ac6d8539e8e74e0931230115f6db75d4ff7 WatchSource:0}: Error finding container 53da545c04e8752f739145afdc125ac6d8539e8e74e0931230115f6db75d4ff7: Status 404 returned error can't find the container with id 53da545c04e8752f739145afdc125ac6d8539e8e74e0931230115f6db75d4ff7
Nov 01 05:02:31.091108 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:31.091070    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e856df5a-5c5e-4421-ae7d-80a760bc72e2-original-pull-secret\") pod \"global-pull-secret-syncer-9pcxf\" (UID: \"e856df5a-5c5e-4421-ae7d-80a760bc72e2\") " pod="kube-system/global-pull-secret-syncer-9pcxf"
Nov 01 05:02:31.091304 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:31.091149    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4321e0ae-e339-40f2-875a-941b1183e766-metrics-certs\") pod \"network-metrics-daemon-267rv\" (UID: \"4321e0ae-e339-40f2-875a-941b1183e766\") " pod="openshift-multus/network-metrics-daemon-267rv"
Nov 01 05:02:31.091372 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:31.091313    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 05:02:31.091428 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:31.091375    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4321e0ae-e339-40f2-875a-941b1183e766-metrics-certs podName:4321e0ae-e339-40f2-875a-941b1183e766 nodeName:}" failed. No retries permitted until 2026-11-01 05:02:33.091356017 +0000 UTC m=+5.292253362 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4321e0ae-e339-40f2-875a-941b1183e766-metrics-certs") pod "network-metrics-daemon-267rv" (UID: "4321e0ae-e339-40f2-875a-941b1183e766") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 05:02:31.091494 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:31.091480    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 05:02:31.091580 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:31.091535    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e856df5a-5c5e-4421-ae7d-80a760bc72e2-original-pull-secret podName:e856df5a-5c5e-4421-ae7d-80a760bc72e2 nodeName:}" failed. No retries permitted until 2026-11-01 05:02:33.091522552 +0000 UTC m=+5.292419887 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e856df5a-5c5e-4421-ae7d-80a760bc72e2-original-pull-secret") pod "global-pull-secret-syncer-9pcxf" (UID: "e856df5a-5c5e-4421-ae7d-80a760bc72e2") : object "kube-system"/"original-pull-secret" not registered
Nov 01 05:02:31.192144 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:31.192103    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffjr4\" (UniqueName: \"kubernetes.io/projected/bcc64278-da5e-4525-90bf-abddde870d70-kube-api-access-ffjr4\") pod \"network-check-target-sr4n8\" (UID: \"bcc64278-da5e-4525-90bf-abddde870d70\") " pod="openshift-network-diagnostics/network-check-target-sr4n8"
Nov 01 05:02:31.192765 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:31.192262    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 05:02:31.192765 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:31.192288    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 05:02:31.192765 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:31.192302    2571 projected.go:196] Error preparing data for projected volume kube-api-access-ffjr4 for pod openshift-network-diagnostics/network-check-target-sr4n8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 05:02:31.192765 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:31.192365    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bcc64278-da5e-4525-90bf-abddde870d70-kube-api-access-ffjr4 podName:bcc64278-da5e-4525-90bf-abddde870d70 nodeName:}" failed. No retries permitted until 2026-11-01 05:02:33.19234117 +0000 UTC m=+5.393238514 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-ffjr4" (UniqueName: "kubernetes.io/projected/bcc64278-da5e-4525-90bf-abddde870d70-kube-api-access-ffjr4") pod "network-check-target-sr4n8" (UID: "bcc64278-da5e-4525-90bf-abddde870d70") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 05:02:31.307406 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:31.307372    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sr4n8"
Nov 01 05:02:31.307607 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:31.307420    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9pcxf"
Nov 01 05:02:31.307607 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:31.307537    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sr4n8" podUID="bcc64278-da5e-4525-90bf-abddde870d70"
Nov 01 05:02:31.307736 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:31.307635    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9pcxf" podUID="e856df5a-5c5e-4421-ae7d-80a760bc72e2"
Nov 01 05:02:31.307736 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:31.307718    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-267rv"
Nov 01 05:02:31.307845 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:31.307806    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-267rv" podUID="4321e0ae-e339-40f2-875a-941b1183e766"
Nov 01 05:02:31.318578 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:31.318542    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hzr4h" event={"ID":"cca47ebc-569a-4558-af41-5e3755605b9d","Type":"ContainerStarted","Data":"4eff5ab2517b779c97c79777f957957fe446c9b09aba3853080f6cc1edf21822"}
Nov 01 05:02:31.320734 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:31.320707    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2982h" event={"ID":"7e9cccdd-6c18-4a60-837f-4907411a1c44","Type":"ContainerStarted","Data":"e4e554abfa6a31cdecc3e57264ae7ce20607c5d8b75e65f9b6d2bdd3c3d435e9"}
Nov 01 05:02:31.323922 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:31.323885    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5cv94" event={"ID":"972b129f-0166-4705-a71c-09ee0e7a91b9","Type":"ContainerStarted","Data":"0a973929be5cebc219b720e4cbc988c2101c4d3885f1252d602d91ece8d10c9b"}
Nov 01 05:02:31.325671 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:31.325645    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6kb4c" event={"ID":"0e24f687-655b-4edb-8901-35412d3fc79b","Type":"ContainerStarted","Data":"53da545c04e8752f739145afdc125ac6d8539e8e74e0931230115f6db75d4ff7"}
Nov 01 05:02:31.327849 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:31.327775    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zm57v" event={"ID":"9752ded2-725f-4035-b374-602e4b4eb576","Type":"ContainerStarted","Data":"c2dfa59500c789ca4744842d55b7f0abfa7266b92f0009d18d6fc18ed125502a"}
Nov 01 05:02:31.332503 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:31.329390    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-d8jfj" event={"ID":"6149f269-49c2-4572-8f6c-fe01e9f0dcbf","Type":"ContainerStarted","Data":"5bc3adeb1375285df89458f45d2042ba303bd03e91b726e1d832adcc059030f9"}
Nov 01 05:02:31.335610 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:31.335583    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ddc5d" event={"ID":"a877f895-a4cf-404d-87e4-440d4b73a870","Type":"ContainerStarted","Data":"fa82b731dbe7fc0fedf4b1e1fc8c3c6dca92546a0a2b97db3d83e7f320bc0bd3"}
Nov 01 05:02:31.569087 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:31.568990    2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 04:57:30 +0000 UTC" deadline="2028-05-25 05:40:52.239594667 +0000 UTC"
Nov 01 05:02:31.569087 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:31.569037    2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13704h38m20.670563481s"
Nov 01 05:02:33.114447 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:33.113996    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4321e0ae-e339-40f2-875a-941b1183e766-metrics-certs\") pod \"network-metrics-daemon-267rv\" (UID: \"4321e0ae-e339-40f2-875a-941b1183e766\") " pod="openshift-multus/network-metrics-daemon-267rv"
Nov 01 05:02:33.114447 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:33.114080    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e856df5a-5c5e-4421-ae7d-80a760bc72e2-original-pull-secret\") pod \"global-pull-secret-syncer-9pcxf\" (UID: \"e856df5a-5c5e-4421-ae7d-80a760bc72e2\") " pod="kube-system/global-pull-secret-syncer-9pcxf"
Nov 01 05:02:33.114447 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:33.114235    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 05:02:33.114447 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:33.114301    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e856df5a-5c5e-4421-ae7d-80a760bc72e2-original-pull-secret podName:e856df5a-5c5e-4421-ae7d-80a760bc72e2 nodeName:}" failed. No retries permitted until 2026-11-01 05:02:37.114282052 +0000 UTC m=+9.315179406 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e856df5a-5c5e-4421-ae7d-80a760bc72e2-original-pull-secret") pod "global-pull-secret-syncer-9pcxf" (UID: "e856df5a-5c5e-4421-ae7d-80a760bc72e2") : object "kube-system"/"original-pull-secret" not registered
Nov 01 05:02:33.114447 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:33.114375    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 05:02:33.114447 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:33.114413    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4321e0ae-e339-40f2-875a-941b1183e766-metrics-certs podName:4321e0ae-e339-40f2-875a-941b1183e766 nodeName:}" failed. No retries permitted until 2026-11-01 05:02:37.114399112 +0000 UTC m=+9.315296457 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4321e0ae-e339-40f2-875a-941b1183e766-metrics-certs") pod "network-metrics-daemon-267rv" (UID: "4321e0ae-e339-40f2-875a-941b1183e766") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 05:02:33.215331 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:33.214931    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffjr4\" (UniqueName: \"kubernetes.io/projected/bcc64278-da5e-4525-90bf-abddde870d70-kube-api-access-ffjr4\") pod \"network-check-target-sr4n8\" (UID: \"bcc64278-da5e-4525-90bf-abddde870d70\") " pod="openshift-network-diagnostics/network-check-target-sr4n8"
Nov 01 05:02:33.215331 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:33.215265    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 05:02:33.215331 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:33.215291    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 05:02:33.215331 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:33.215305    2571 projected.go:196] Error preparing data for projected volume kube-api-access-ffjr4 for pod openshift-network-diagnostics/network-check-target-sr4n8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 05:02:33.215764 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:33.215383    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bcc64278-da5e-4525-90bf-abddde870d70-kube-api-access-ffjr4 podName:bcc64278-da5e-4525-90bf-abddde870d70 nodeName:}" failed. No retries permitted until 2026-11-01 05:02:37.215362563 +0000 UTC m=+9.416259910 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-ffjr4" (UniqueName: "kubernetes.io/projected/bcc64278-da5e-4525-90bf-abddde870d70-kube-api-access-ffjr4") pod "network-check-target-sr4n8" (UID: "bcc64278-da5e-4525-90bf-abddde870d70") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 05:02:33.307640 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:33.307594    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sr4n8"
Nov 01 05:02:33.307827 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:33.307725    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sr4n8" podUID="bcc64278-da5e-4525-90bf-abddde870d70"
Nov 01 05:02:33.307827 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:33.307798    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9pcxf"
Nov 01 05:02:33.307948 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:33.307875    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9pcxf" podUID="e856df5a-5c5e-4421-ae7d-80a760bc72e2"
Nov 01 05:02:33.307948 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:33.307942    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-267rv"
Nov 01 05:02:33.308053 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:33.308030    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-267rv" podUID="4321e0ae-e339-40f2-875a-941b1183e766"
Nov 01 05:02:33.354592 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:33.351602    2571 generic.go:356] "Generic (PLEG): container finished" podID="a9bb6df5-0993-4640-81cc-52426b72fb3e" containerID="3e79c22b5e5c930ffb647b4821134e4ce2e2a3d76b0f1199fcb6ed2ca19648a2" exitCode=0
Nov 01 05:02:33.354592 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:33.351657    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r4nh2" event={"ID":"a9bb6df5-0993-4640-81cc-52426b72fb3e","Type":"ContainerDied","Data":"3e79c22b5e5c930ffb647b4821134e4ce2e2a3d76b0f1199fcb6ed2ca19648a2"}
Nov 01 05:02:34.358951 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:34.358823    2571 generic.go:356] "Generic (PLEG): container finished" podID="99a2e3eae38c794f2284a61da069e044" containerID="0f8c39ab12f2e6999375d4135fb54dc15e3a611b01e27d18067ee26e6ca2db1c" exitCode=0
Nov 01 05:02:34.358951 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:34.358882    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-92.us-west-1.compute.internal" event={"ID":"99a2e3eae38c794f2284a61da069e044","Type":"ContainerDied","Data":"0f8c39ab12f2e6999375d4135fb54dc15e3a611b01e27d18067ee26e6ca2db1c"}
Nov 01 05:02:35.308285 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:35.307201    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-267rv"
Nov 01 05:02:35.308285 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:35.307343    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-267rv" podUID="4321e0ae-e339-40f2-875a-941b1183e766"
Nov 01 05:02:35.308285 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:35.307775    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sr4n8"
Nov 01 05:02:35.308285 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:35.307879    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sr4n8" podUID="bcc64278-da5e-4525-90bf-abddde870d70"
Nov 01 05:02:35.308285 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:35.307950    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9pcxf"
Nov 01 05:02:35.308285 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:35.308030    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9pcxf" podUID="e856df5a-5c5e-4421-ae7d-80a760bc72e2"
Nov 01 05:02:35.373442 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:35.373389    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-92.us-west-1.compute.internal" event={"ID":"99a2e3eae38c794f2284a61da069e044","Type":"ContainerStarted","Data":"572fd80986c4e0d7538a35b388de3d4d19093ff7b75b47ae0da8d8cd5d73b317"}
Nov 01 05:02:35.379287 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:35.379243    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-92.us-west-1.compute.internal" event={"ID":"0742a37cb8250b02dcdb2d2301dda7a1","Type":"ContainerStarted","Data":"d516f6ed608b9ff59d17f0b67627c9272357bfe2fd97511db96b9548fa682736"}
Nov 01 05:02:35.627631 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:35.627505    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-92.us-west-1.compute.internal" podStartSLOduration=6.627484633 podStartE2EDuration="6.627484633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 05:02:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:02:35.501347983 +0000 UTC m=+7.702245339" watchObservedRunningTime="2026-11-01 05:02:35.627484633 +0000 UTC m=+7.828381989"
Nov 01 05:02:36.396696 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:36.396053    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vrbks" event={"ID":"d454e879-03a2-4fb9-8223-79a1aa9d85d2","Type":"ContainerStarted","Data":"c35618c9ff78bb41d0d9a491caa2be7cb9bb054b90cb01b4cc2e33af6015e13b"}
Nov 01 05:02:36.405114 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:36.404647    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hzr4h" event={"ID":"cca47ebc-569a-4558-af41-5e3755605b9d","Type":"ContainerStarted","Data":"1ece5ab602f345ba62be96519f3d7f4db3ef5e05c97e3c36fe9972d942fb7ef3"}
Nov 01 05:02:36.522380 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:36.522318    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-92.us-west-1.compute.internal" podStartSLOduration=7.522272408 podStartE2EDuration="7.522272408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 05:02:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:02:35.628465733 +0000 UTC m=+7.829363090" watchObservedRunningTime="2026-11-01 05:02:36.522272408 +0000 UTC m=+8.723169764"
Nov 01 05:02:36.522610 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:36.522512    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-vrbks" podStartSLOduration=2.295510393 podStartE2EDuration="7.522505099s" totalImagesPullingTime="5.226994706s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 05:02:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:02:36.521876596 +0000 UTC m=+8.722773952" watchObservedRunningTime="2026-11-01 05:02:36.522505099 +0000 UTC m=+8.723402454"
Nov 01 05:02:37.152954 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:37.152868    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4321e0ae-e339-40f2-875a-941b1183e766-metrics-certs\") pod \"network-metrics-daemon-267rv\" (UID: \"4321e0ae-e339-40f2-875a-941b1183e766\") " pod="openshift-multus/network-metrics-daemon-267rv"
Nov 01 05:02:37.152954 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:37.152934    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e856df5a-5c5e-4421-ae7d-80a760bc72e2-original-pull-secret\") pod \"global-pull-secret-syncer-9pcxf\" (UID: \"e856df5a-5c5e-4421-ae7d-80a760bc72e2\") " pod="kube-system/global-pull-secret-syncer-9pcxf"
Nov 01 05:02:37.153191 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:37.153058    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 05:02:37.153191 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:37.153115    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e856df5a-5c5e-4421-ae7d-80a760bc72e2-original-pull-secret podName:e856df5a-5c5e-4421-ae7d-80a760bc72e2 nodeName:}" failed. No retries permitted until 2026-11-01 05:02:45.153102936 +0000 UTC m=+17.354000267 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e856df5a-5c5e-4421-ae7d-80a760bc72e2-original-pull-secret") pod "global-pull-secret-syncer-9pcxf" (UID: "e856df5a-5c5e-4421-ae7d-80a760bc72e2") : object "kube-system"/"original-pull-secret" not registered
Nov 01 05:02:37.153503 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:37.153483    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 05:02:37.153592 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:37.153554    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4321e0ae-e339-40f2-875a-941b1183e766-metrics-certs podName:4321e0ae-e339-40f2-875a-941b1183e766 nodeName:}" failed. No retries permitted until 2026-11-01 05:02:45.153537556 +0000 UTC m=+17.354434911 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4321e0ae-e339-40f2-875a-941b1183e766-metrics-certs") pod "network-metrics-daemon-267rv" (UID: "4321e0ae-e339-40f2-875a-941b1183e766") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 05:02:37.254042 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:37.253996    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffjr4\" (UniqueName: \"kubernetes.io/projected/bcc64278-da5e-4525-90bf-abddde870d70-kube-api-access-ffjr4\") pod \"network-check-target-sr4n8\" (UID: \"bcc64278-da5e-4525-90bf-abddde870d70\") " pod="openshift-network-diagnostics/network-check-target-sr4n8"
Nov 01 05:02:37.254251 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:37.254226    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 05:02:37.254348 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:37.254257    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 05:02:37.254348 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:37.254273    2571 projected.go:196] Error preparing data for projected volume kube-api-access-ffjr4 for pod openshift-network-diagnostics/network-check-target-sr4n8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 05:02:37.254348 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:37.254336    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bcc64278-da5e-4525-90bf-abddde870d70-kube-api-access-ffjr4 podName:bcc64278-da5e-4525-90bf-abddde870d70 nodeName:}" failed. No retries permitted until 2026-11-01 05:02:45.254315428 +0000 UTC m=+17.455212780 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-ffjr4" (UniqueName: "kubernetes.io/projected/bcc64278-da5e-4525-90bf-abddde870d70-kube-api-access-ffjr4") pod "network-check-target-sr4n8" (UID: "bcc64278-da5e-4525-90bf-abddde870d70") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 05:02:37.307419 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:37.307384    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sr4n8"
Nov 01 05:02:37.307624 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:37.307597    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sr4n8" podUID="bcc64278-da5e-4525-90bf-abddde870d70"
Nov 01 05:02:37.307704 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:37.307653    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-267rv"
Nov 01 05:02:37.307704 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:37.307694    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9pcxf"
Nov 01 05:02:37.307842 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:37.307819    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-267rv" podUID="4321e0ae-e339-40f2-875a-941b1183e766"
Nov 01 05:02:37.307963 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:37.307939    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9pcxf" podUID="e856df5a-5c5e-4421-ae7d-80a760bc72e2"
Nov 01 05:02:37.418656 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:37.418215    2571 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-hzr4h"
Nov 01 05:02:37.419142 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:37.418665    2571 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-hzr4h"
Nov 01 05:02:37.545285 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:37.545223    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-hzr4h" podStartSLOduration=3.733239459 podStartE2EDuration="8.545201245s" totalImagesPullingTime="4.811961786s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:02:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:02:36.665957785 +0000 UTC m=+8.866855141" watchObservedRunningTime="2026-11-01 05:02:37.545201245 +0000 UTC m=+9.746098600"
Nov 01 05:02:38.083328 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:38.083293    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-hzr4h"
Nov 01 05:02:38.084569 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:38.084356    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-hzr4h"
Nov 01 05:02:38.413471 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:38.413343    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6kb4c" event={"ID":"0e24f687-655b-4edb-8901-35412d3fc79b","Type":"ContainerStarted","Data":"a403b2cbd7768fe70ef3169ffeaa843580a3d852ba5c10ce519dd1b3eaa9c1f5"}
Nov 01 05:02:39.307720 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:39.307645    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sr4n8"
Nov 01 05:02:39.307720 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:39.307692    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9pcxf"
Nov 01 05:02:39.308174 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:39.307782    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sr4n8" podUID="bcc64278-da5e-4525-90bf-abddde870d70"
Nov 01 05:02:39.308174 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:39.307652    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-267rv"
Nov 01 05:02:39.308448 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:39.308278    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-267rv" podUID="4321e0ae-e339-40f2-875a-941b1183e766"
Nov 01 05:02:39.308448 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:39.308400    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9pcxf" podUID="e856df5a-5c5e-4421-ae7d-80a760bc72e2"
Nov 01 05:02:39.417976 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:39.417934    2571 generic.go:356] "Generic (PLEG): container finished" podID="a9bb6df5-0993-4640-81cc-52426b72fb3e" containerID="a7368cd51823aeba5ed2d567748e96aea78316e7f23b8bad5753e8d444b65622" exitCode=0
Nov 01 05:02:39.419044 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:39.419017    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r4nh2" event={"ID":"a9bb6df5-0993-4640-81cc-52426b72fb3e","Type":"ContainerDied","Data":"a7368cd51823aeba5ed2d567748e96aea78316e7f23b8bad5753e8d444b65622"}
Nov 01 05:02:39.802799 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:39.802774    2571 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 01 05:02:40.232575 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:40.232386    2571 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T05:02:39.802792688Z","UUID":"b950f281-c78c-45d3-9c05-b0b9dc97cc87","Handler":null,"Name":"","Endpoint":""}
Nov 01 05:02:40.234059 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:40.234039    2571 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 01 05:02:40.234174 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:40.234069    2571 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 01 05:02:40.423400 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:40.422146    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5cv94" event={"ID":"972b129f-0166-4705-a71c-09ee0e7a91b9","Type":"ContainerStarted","Data":"40e188f9c166394de6dfbcceceabaccc3802276ebb6e06edd31bb74d6fbf9966"}
Nov 01 05:02:40.425352 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:40.425320    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6kb4c" event={"ID":"0e24f687-655b-4edb-8901-35412d3fc79b","Type":"ContainerStarted","Data":"f397dcf8aaccf3cccb20b88a2dfb07f33b47f65df3b5fa9c66291457caae2875"}
Nov 01 05:02:40.426815 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:40.426789    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ddc5d" event={"ID":"a877f895-a4cf-404d-87e4-440d4b73a870","Type":"ContainerStarted","Data":"9a601c4add0475977747ca7704a63134933098ac9d6ec880f0fb36656cfa9c7d"}
Nov 01 05:02:40.542362 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:40.542061    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-5cv94" podStartSLOduration=2.772699362 podStartE2EDuration="11.542040201s" totalImagesPullingTime="8.769340839s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 05:02:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:02:40.541481309 +0000 UTC m=+12.742378660" watchObservedRunningTime="2026-11-01 05:02:40.542040201 +0000 UTC m=+12.742937552"
Nov 01 05:02:41.308237 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:41.308150    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sr4n8"
Nov 01 05:02:41.308383 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:41.308165    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9pcxf"
Nov 01 05:02:41.308383 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:41.308268    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-267rv"
Nov 01 05:02:41.308383 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:41.308265    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sr4n8" podUID="bcc64278-da5e-4525-90bf-abddde870d70"
Nov 01 05:02:41.308383 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:41.308342    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9pcxf" podUID="e856df5a-5c5e-4421-ae7d-80a760bc72e2"
Nov 01 05:02:41.308535 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:41.308427    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-267rv" podUID="4321e0ae-e339-40f2-875a-941b1183e766"
Nov 01 05:02:41.431141 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:41.431103    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-d8jfj" event={"ID":"6149f269-49c2-4572-8f6c-fe01e9f0dcbf","Type":"ContainerStarted","Data":"ae98ca1acffc5342fbd76e703f162d37baabef4b56bb0cb46697ad1ef4a0fc59"}
Nov 01 05:02:41.435101 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:41.435071    2571 generic.go:356] "Generic (PLEG): container finished" podID="a9bb6df5-0993-4640-81cc-52426b72fb3e" containerID="1e20a3e84f8ad12701497600c153da060cdd75e5a8e95c24363a5a7531206de5" exitCode=0
Nov 01 05:02:41.435218 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:41.435157    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r4nh2" event={"ID":"a9bb6df5-0993-4640-81cc-52426b72fb3e","Type":"ContainerDied","Data":"1e20a3e84f8ad12701497600c153da060cdd75e5a8e95c24363a5a7531206de5"}
Nov 01 05:02:41.546749 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:41.546686    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-ddc5d" podStartSLOduration=3.668117666 podStartE2EDuration="12.546669256s" totalImagesPullingTime="8.87855159s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 05:02:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:02:40.65791755 +0000 UTC m=+12.858814902" watchObservedRunningTime="2026-11-01 05:02:41.546669256 +0000 UTC m=+13.747566608"
Nov 01 05:02:41.546901 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:41.546847    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-d8jfj" podStartSLOduration=3.569266635 podStartE2EDuration="12.546837074s" totalImagesPullingTime="8.977570439s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 05:02:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:02:41.546470588 +0000 UTC m=+13.747367936" watchObservedRunningTime="2026-11-01 05:02:41.546837074 +0000 UTC m=+13.747734426"
Nov 01 05:02:42.441610 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:42.440586    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6kb4c" event={"ID":"0e24f687-655b-4edb-8901-35412d3fc79b","Type":"ContainerStarted","Data":"751aaf271f0cc3cf3e02c9baa4f54d430bc859aa5a29b5786f4729d21f32d151"}
Nov 01 05:02:42.442539 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:42.442508    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zm57v" event={"ID":"9752ded2-725f-4035-b374-602e4b4eb576","Type":"ContainerStarted","Data":"e0fb63ef2677bcbbf3facf595b402d8c9e60a805a971fb4258c24d130cb35149"}
Nov 01 05:02:42.559574 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:42.559524    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6kb4c" podStartSLOduration=2.901726031 podStartE2EDuration="13.559510518s" totalImagesPullingTime="10.657784487s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 05:02:29 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:02:42.559267331 +0000 UTC m=+14.760164705" watchObservedRunningTime="2026-11-01 05:02:42.559510518 +0000 UTC m=+14.760407868"
Nov 01 05:02:42.677562 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:42.677509    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-zm57v" podStartSLOduration=2.440731447 podStartE2EDuration="13.677487884s" totalImagesPullingTime="11.236756437s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 05:02:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:02:42.677013505 +0000 UTC m=+14.877910880" watchObservedRunningTime="2026-11-01 05:02:42.677487884 +0000 UTC m=+14.878385236"
Nov 01 05:02:43.308035 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:43.307955    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sr4n8"
Nov 01 05:02:43.308197 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:43.308033    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-267rv"
Nov 01 05:02:43.308197 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:43.308159    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9pcxf"
Nov 01 05:02:43.308197 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:43.308169    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sr4n8" podUID="bcc64278-da5e-4525-90bf-abddde870d70"
Nov 01 05:02:43.308363 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:43.308344    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9pcxf" podUID="e856df5a-5c5e-4421-ae7d-80a760bc72e2"
Nov 01 05:02:43.308478 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:43.308436    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-267rv" podUID="4321e0ae-e339-40f2-875a-941b1183e766"
Nov 01 05:02:43.451131 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:43.451099    2571 generic.go:356] "Generic (PLEG): container finished" podID="a9bb6df5-0993-4640-81cc-52426b72fb3e" containerID="c6e7595015fd306e4bcd4b8c1de3981d99de3144748471519ad41938bb8ec6cd" exitCode=0
Nov 01 05:02:43.451826 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:43.451185    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r4nh2" event={"ID":"a9bb6df5-0993-4640-81cc-52426b72fb3e","Type":"ContainerDied","Data":"c6e7595015fd306e4bcd4b8c1de3981d99de3144748471519ad41938bb8ec6cd"}
Nov 01 05:02:45.222077 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:45.222039    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4321e0ae-e339-40f2-875a-941b1183e766-metrics-certs\") pod \"network-metrics-daemon-267rv\" (UID: \"4321e0ae-e339-40f2-875a-941b1183e766\") " pod="openshift-multus/network-metrics-daemon-267rv"
Nov 01 05:02:45.222764 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:45.222093    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e856df5a-5c5e-4421-ae7d-80a760bc72e2-original-pull-secret\") pod \"global-pull-secret-syncer-9pcxf\" (UID: \"e856df5a-5c5e-4421-ae7d-80a760bc72e2\") " pod="kube-system/global-pull-secret-syncer-9pcxf"
Nov 01 05:02:45.222764 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:45.222188    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 05:02:45.222764 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:45.222201    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 05:02:45.222764 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:45.222251    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e856df5a-5c5e-4421-ae7d-80a760bc72e2-original-pull-secret podName:e856df5a-5c5e-4421-ae7d-80a760bc72e2 nodeName:}" failed. No retries permitted until 2026-11-01 05:03:01.222233489 +0000 UTC m=+33.423130854 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e856df5a-5c5e-4421-ae7d-80a760bc72e2-original-pull-secret") pod "global-pull-secret-syncer-9pcxf" (UID: "e856df5a-5c5e-4421-ae7d-80a760bc72e2") : object "kube-system"/"original-pull-secret" not registered
Nov 01 05:02:45.222764 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:45.222270    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4321e0ae-e339-40f2-875a-941b1183e766-metrics-certs podName:4321e0ae-e339-40f2-875a-941b1183e766 nodeName:}" failed. No retries permitted until 2026-11-01 05:03:01.222262608 +0000 UTC m=+33.423159940 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4321e0ae-e339-40f2-875a-941b1183e766-metrics-certs") pod "network-metrics-daemon-267rv" (UID: "4321e0ae-e339-40f2-875a-941b1183e766") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 05:02:45.307525 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:45.307492    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-267rv"
Nov 01 05:02:45.307525 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:45.307525    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9pcxf"
Nov 01 05:02:45.307768 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:45.307539    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sr4n8"
Nov 01 05:02:45.307768 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:45.307640    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-267rv" podUID="4321e0ae-e339-40f2-875a-941b1183e766"
Nov 01 05:02:45.307768 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:45.307725    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9pcxf" podUID="e856df5a-5c5e-4421-ae7d-80a760bc72e2"
Nov 01 05:02:45.307884 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:45.307810    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sr4n8" podUID="bcc64278-da5e-4525-90bf-abddde870d70"
Nov 01 05:02:45.322724 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:45.322701    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffjr4\" (UniqueName: \"kubernetes.io/projected/bcc64278-da5e-4525-90bf-abddde870d70-kube-api-access-ffjr4\") pod \"network-check-target-sr4n8\" (UID: \"bcc64278-da5e-4525-90bf-abddde870d70\") " pod="openshift-network-diagnostics/network-check-target-sr4n8"
Nov 01 05:02:45.322855 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:45.322818    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 05:02:45.322855 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:45.322835    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 05:02:45.322855 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:45.322844    2571 projected.go:196] Error preparing data for projected volume kube-api-access-ffjr4 for pod openshift-network-diagnostics/network-check-target-sr4n8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 05:02:45.322981 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:45.322903    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bcc64278-da5e-4525-90bf-abddde870d70-kube-api-access-ffjr4 podName:bcc64278-da5e-4525-90bf-abddde870d70 nodeName:}" failed. No retries permitted until 2026-11-01 05:03:01.322886074 +0000 UTC m=+33.523783419 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-ffjr4" (UniqueName: "kubernetes.io/projected/bcc64278-da5e-4525-90bf-abddde870d70-kube-api-access-ffjr4") pod "network-check-target-sr4n8" (UID: "bcc64278-da5e-4525-90bf-abddde870d70") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 05:02:46.462613 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:46.462588    2571 generic.go:356] "Generic (PLEG): container finished" podID="a9bb6df5-0993-4640-81cc-52426b72fb3e" containerID="478db2d69005981efb029034aec26cb2146045e035d1f79216434b4306c98e2e" exitCode=0
Nov 01 05:02:46.462980 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:46.462683    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r4nh2" event={"ID":"a9bb6df5-0993-4640-81cc-52426b72fb3e","Type":"ContainerDied","Data":"478db2d69005981efb029034aec26cb2146045e035d1f79216434b4306c98e2e"}
Nov 01 05:02:46.464799 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:46.464776    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2982h" event={"ID":"7e9cccdd-6c18-4a60-837f-4907411a1c44","Type":"ContainerStarted","Data":"5191566e5cf697a81ec7b7a64e571e58e5e802eb49b5508d43dc505889cfec33"}
Nov 01 05:02:46.464881 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:46.464841    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2982h" event={"ID":"7e9cccdd-6c18-4a60-837f-4907411a1c44","Type":"ContainerStarted","Data":"b1b8882d7773db979e6b8f0fe6c0fe3b36a69b536eb32d01883c7a29b2576d35"}
Nov 01 05:02:47.307333 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:47.307150    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-267rv"
Nov 01 05:02:47.307503 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:47.307210    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9pcxf"
Nov 01 05:02:47.307503 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:47.307414    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-267rv" podUID="4321e0ae-e339-40f2-875a-941b1183e766"
Nov 01 05:02:47.307503 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:47.307488    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9pcxf" podUID="e856df5a-5c5e-4421-ae7d-80a760bc72e2"
Nov 01 05:02:47.307503 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:47.307217    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sr4n8"
Nov 01 05:02:47.307639 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:47.307554    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sr4n8" podUID="bcc64278-da5e-4525-90bf-abddde870d70"
Nov 01 05:02:47.469596 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:47.469558    2571 generic.go:356] "Generic (PLEG): container finished" podID="a9bb6df5-0993-4640-81cc-52426b72fb3e" containerID="dee5a815f5eaa4269c1bf5d730227d7a3b7f6f9ff38eeb573337ac08fd896dac" exitCode=0
Nov 01 05:02:47.469953 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:47.469654    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r4nh2" event={"ID":"a9bb6df5-0993-4640-81cc-52426b72fb3e","Type":"ContainerDied","Data":"dee5a815f5eaa4269c1bf5d730227d7a3b7f6f9ff38eeb573337ac08fd896dac"}
Nov 01 05:02:47.472723 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:47.472703    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2982h" event={"ID":"7e9cccdd-6c18-4a60-837f-4907411a1c44","Type":"ContainerStarted","Data":"9f49146ae79bb6ed5294a2e43a5602b2d83d93c5a0b826161483445a029ab51a"}
Nov 01 05:02:47.472785 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:47.472758    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2982h" event={"ID":"7e9cccdd-6c18-4a60-837f-4907411a1c44","Type":"ContainerStarted","Data":"bc1ac8e51c4d85cfe53d38c3df12d59458090b50a3a1cf2d71cc3253bf4c32b5"}
Nov 01 05:02:47.472829 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:47.472785    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2982h" event={"ID":"7e9cccdd-6c18-4a60-837f-4907411a1c44","Type":"ContainerStarted","Data":"05b72664a53f5f544b95422140b9509039364f10204ced332ad8e97cdb4feab6"}
Nov 01 05:02:47.472829 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:47.472814    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2982h" event={"ID":"7e9cccdd-6c18-4a60-837f-4907411a1c44","Type":"ContainerStarted","Data":"9c03a0979ae67007f092a106d73b326d40f5bfb05cbb8cd7e0f8bd7e0ce38c50"}
Nov 01 05:02:48.478232 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:48.478197    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r4nh2" event={"ID":"a9bb6df5-0993-4640-81cc-52426b72fb3e","Type":"ContainerStarted","Data":"1cf4201bf75a05448240b18359c4fcee7eb1b074afdce7117eef3439023edbd4"}
Nov 01 05:02:48.601434 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:48.601385    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-r4nh2" podStartSLOduration=6.705469843 podStartE2EDuration="19.601373328s" totalImagesPullingTime="12.895903485s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 05:02:29 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:02:48.600817795 +0000 UTC m=+20.801715148" watchObservedRunningTime="2026-11-01 05:02:48.601373328 +0000 UTC m=+20.802270678"
Nov 01 05:02:49.307504 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:49.307473    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sr4n8"
Nov 01 05:02:49.307658 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:49.307472    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9pcxf"
Nov 01 05:02:49.307658 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:49.307563    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sr4n8" podUID="bcc64278-da5e-4525-90bf-abddde870d70"
Nov 01 05:02:49.307658 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:49.307581    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-267rv"
Nov 01 05:02:49.307744 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:49.307673    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9pcxf" podUID="e856df5a-5c5e-4421-ae7d-80a760bc72e2"
Nov 01 05:02:49.307792 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:49.307774    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-267rv" podUID="4321e0ae-e339-40f2-875a-941b1183e766"
Nov 01 05:02:49.483424 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:49.483398    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2982h" event={"ID":"7e9cccdd-6c18-4a60-837f-4907411a1c44","Type":"ContainerStarted","Data":"d44b0f461b5f7d739ba3130e0e9019d757d08d83b43b893408d10bb5bae31921"}
Nov 01 05:02:51.307619 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:51.307437    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sr4n8"
Nov 01 05:02:51.308280 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:51.307526    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9pcxf"
Nov 01 05:02:51.308280 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:51.307696    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sr4n8" podUID="bcc64278-da5e-4525-90bf-abddde870d70"
Nov 01 05:02:51.308280 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:51.307530    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-267rv"
Nov 01 05:02:51.308280 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:51.307769    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9pcxf" podUID="e856df5a-5c5e-4421-ae7d-80a760bc72e2"
Nov 01 05:02:51.308280 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:51.307905    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-267rv" podUID="4321e0ae-e339-40f2-875a-941b1183e766"
Nov 01 05:02:51.493803 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:51.493772    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2982h" event={"ID":"7e9cccdd-6c18-4a60-837f-4907411a1c44","Type":"ContainerStarted","Data":"979b5a2ab25c62092ccbc82e4d0f07a90de6463ce737fa0b6741bb9def82f200"}
Nov 01 05:02:51.494049 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:51.494030    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:02:51.494116 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:51.494105    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:02:51.494160 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:51.494146    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:02:51.509967 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:51.509943    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:02:51.510112 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:51.510055    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:02:51.774594 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:51.774550    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-2982h" podStartSLOduration=7.108788274 podStartE2EDuration="22.774533489s" totalImagesPullingTime="15.665745215s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 05:02:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:02:51.723592529 +0000 UTC m=+23.924489879" watchObservedRunningTime="2026-11-01 05:02:51.774533489 +0000 UTC m=+23.975430840"
Nov 01 05:02:53.307787 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:53.307753    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sr4n8"
Nov 01 05:02:53.308214 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:53.307811    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-267rv"
Nov 01 05:02:53.308214 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:53.307899    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9pcxf"
Nov 01 05:02:53.308214 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:53.307901    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sr4n8" podUID="bcc64278-da5e-4525-90bf-abddde870d70"
Nov 01 05:02:53.308214 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:53.307990    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-267rv" podUID="4321e0ae-e339-40f2-875a-941b1183e766"
Nov 01 05:02:53.308214 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:53.308040    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9pcxf" podUID="e856df5a-5c5e-4421-ae7d-80a760bc72e2"
Nov 01 05:02:53.606023 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:53.605941    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9pcxf"]
Nov 01 05:02:53.606197 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:53.606045    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9pcxf"
Nov 01 05:02:53.606197 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:53.606141    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9pcxf" podUID="e856df5a-5c5e-4421-ae7d-80a760bc72e2"
Nov 01 05:02:53.609235 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:53.609212    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-sr4n8"]
Nov 01 05:02:53.609360 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:53.609298    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sr4n8"
Nov 01 05:02:53.609422 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:53.609394    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sr4n8" podUID="bcc64278-da5e-4525-90bf-abddde870d70"
Nov 01 05:02:53.609829 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:53.609810    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-267rv"]
Nov 01 05:02:53.609891 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:53.609881    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-267rv"
Nov 01 05:02:53.609979 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:53.609962    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-267rv" podUID="4321e0ae-e339-40f2-875a-941b1183e766"
Nov 01 05:02:53.911895 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:53.911862    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:02:54.127790 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:54.127757    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:02:54.356639 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:54.356554    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:02:54.529264 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:54.529229    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:02:55.009037 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:55.009003    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:02:55.184119 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:55.184091    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:02:55.307173 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:55.307148    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9pcxf"
Nov 01 05:02:55.307173 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:55.307167    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-267rv"
Nov 01 05:02:55.307341 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:55.307148    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sr4n8"
Nov 01 05:02:55.307341 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:55.307270    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9pcxf" podUID="e856df5a-5c5e-4421-ae7d-80a760bc72e2"
Nov 01 05:02:55.307341 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:55.307321    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sr4n8" podUID="bcc64278-da5e-4525-90bf-abddde870d70"
Nov 01 05:02:55.307447 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:55.307408    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-267rv" podUID="4321e0ae-e339-40f2-875a-941b1183e766"
Nov 01 05:02:56.397802 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:56.397769    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:02:56.570569 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:56.570531    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:02:57.031652 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:57.031580    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:02:57.264564 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:57.264528    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:02:57.307474 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:57.307397    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-267rv"
Nov 01 05:02:57.307590 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:57.307396    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9pcxf"
Nov 01 05:02:57.307590 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:57.307522    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-267rv" podUID="4321e0ae-e339-40f2-875a-941b1183e766"
Nov 01 05:02:57.307590 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:57.307396    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sr4n8"
Nov 01 05:02:57.307590 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:57.307580    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9pcxf" podUID="e856df5a-5c5e-4421-ae7d-80a760bc72e2"
Nov 01 05:02:57.307708 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:57.307654    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sr4n8" podUID="bcc64278-da5e-4525-90bf-abddde870d70"
Nov 01 05:02:59.307490 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:59.307299    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9pcxf"
Nov 01 05:02:59.307930 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:59.307331    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sr4n8"
Nov 01 05:02:59.307930 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:59.307561    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9pcxf" podUID="e856df5a-5c5e-4421-ae7d-80a760bc72e2"
Nov 01 05:02:59.307930 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:59.307659    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sr4n8" podUID="bcc64278-da5e-4525-90bf-abddde870d70"
Nov 01 05:02:59.307930 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:59.307340    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-267rv"
Nov 01 05:02:59.307930 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:02:59.307765    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-267rv" podUID="4321e0ae-e339-40f2-875a-941b1183e766"
Nov 01 05:02:59.595115 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:59.595040    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:02:59.607668 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:59.607648    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-92.us-west-1.compute.internal" event="NodeReady"
Nov 01 05:02:59.607809 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:59.607795    2571 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 01 05:02:59.704423 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:59.704398    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-g89wb"]
Nov 01 05:02:59.778345 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:59.778322    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-ljs77"]
Nov 01 05:02:59.778487 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:59.778466    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-g89wb"
Nov 01 05:02:59.790506 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:59.790484    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ljs77"]
Nov 01 05:02:59.790635 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:59.790528    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-g89wb"]
Nov 01 05:02:59.790635 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:59.790605    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ljs77"
Nov 01 05:02:59.795434 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:59.795407    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:02:59.831805 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:59.831777    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 01 05:02:59.831883 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:59.831829    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-27c4k\""
Nov 01 05:02:59.831883 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:59.831781    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 01 05:02:59.844754 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:59.844728    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 01 05:02:59.844963 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:59.844947    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-7m49c\""
Nov 01 05:02:59.845027 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:59.844955    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 01 05:02:59.845027 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:59.844982    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 01 05:02:59.932219 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:59.932198    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/af0da37a-a117-49a6-a06e-6cab0e56a40c-cert\") pod \"ingress-canary-ljs77\" (UID: \"af0da37a-a117-49a6-a06e-6cab0e56a40c\") " pod="openshift-ingress-canary/ingress-canary-ljs77"
Nov 01 05:02:59.932321 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:59.932227    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0b0122f6-d51a-4338-a631-056ec9311b69-metrics-tls\") pod \"dns-default-g89wb\" (UID: \"0b0122f6-d51a-4338-a631-056ec9311b69\") " pod="openshift-dns/dns-default-g89wb"
Nov 01 05:02:59.932321 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:59.932264    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b0122f6-d51a-4338-a631-056ec9311b69-tmp-dir\") pod \"dns-default-g89wb\" (UID: \"0b0122f6-d51a-4338-a631-056ec9311b69\") " pod="openshift-dns/dns-default-g89wb"
Nov 01 05:02:59.932406 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:59.932354    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd22d\" (UniqueName: \"kubernetes.io/projected/af0da37a-a117-49a6-a06e-6cab0e56a40c-kube-api-access-hd22d\") pod \"ingress-canary-ljs77\" (UID: \"af0da37a-a117-49a6-a06e-6cab0e56a40c\") " pod="openshift-ingress-canary/ingress-canary-ljs77"
Nov 01 05:02:59.932445 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:59.932411    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0b0122f6-d51a-4338-a631-056ec9311b69-config-volume\") pod \"dns-default-g89wb\" (UID: \"0b0122f6-d51a-4338-a631-056ec9311b69\") " pod="openshift-dns/dns-default-g89wb"
Nov 01 05:02:59.932507 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:02:59.932477    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9njs4\" (UniqueName: \"kubernetes.io/projected/0b0122f6-d51a-4338-a631-056ec9311b69-kube-api-access-9njs4\") pod \"dns-default-g89wb\" (UID: \"0b0122f6-d51a-4338-a631-056ec9311b69\") " pod="openshift-dns/dns-default-g89wb"
Nov 01 05:03:00.032980 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:00.032958    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9njs4\" (UniqueName: \"kubernetes.io/projected/0b0122f6-d51a-4338-a631-056ec9311b69-kube-api-access-9njs4\") pod \"dns-default-g89wb\" (UID: \"0b0122f6-d51a-4338-a631-056ec9311b69\") " pod="openshift-dns/dns-default-g89wb"
Nov 01 05:03:00.033093 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:00.032986    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/af0da37a-a117-49a6-a06e-6cab0e56a40c-cert\") pod \"ingress-canary-ljs77\" (UID: \"af0da37a-a117-49a6-a06e-6cab0e56a40c\") " pod="openshift-ingress-canary/ingress-canary-ljs77"
Nov 01 05:03:00.033093 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:00.033006    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0b0122f6-d51a-4338-a631-056ec9311b69-metrics-tls\") pod \"dns-default-g89wb\" (UID: \"0b0122f6-d51a-4338-a631-056ec9311b69\") " pod="openshift-dns/dns-default-g89wb"
Nov 01 05:03:00.033163 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:00.033142    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b0122f6-d51a-4338-a631-056ec9311b69-tmp-dir\") pod \"dns-default-g89wb\" (UID: \"0b0122f6-d51a-4338-a631-056ec9311b69\") " pod="openshift-dns/dns-default-g89wb"
Nov 01 05:03:00.033233 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:00.033168    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd22d\" (UniqueName: \"kubernetes.io/projected/af0da37a-a117-49a6-a06e-6cab0e56a40c-kube-api-access-hd22d\") pod \"ingress-canary-ljs77\" (UID: \"af0da37a-a117-49a6-a06e-6cab0e56a40c\") " pod="openshift-ingress-canary/ingress-canary-ljs77"
Nov 01 05:03:00.033233 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:00.033217    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0b0122f6-d51a-4338-a631-056ec9311b69-config-volume\") pod \"dns-default-g89wb\" (UID: \"0b0122f6-d51a-4338-a631-056ec9311b69\") " pod="openshift-dns/dns-default-g89wb"
Nov 01 05:03:00.033308 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:03:00.033253    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 05:03:00.033341 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:03:00.033314    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b0122f6-d51a-4338-a631-056ec9311b69-metrics-tls podName:0b0122f6-d51a-4338-a631-056ec9311b69 nodeName:}" failed. No retries permitted until 2026-11-01 05:03:00.533294917 +0000 UTC m=+32.734192248 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0b0122f6-d51a-4338-a631-056ec9311b69-metrics-tls") pod "dns-default-g89wb" (UID: "0b0122f6-d51a-4338-a631-056ec9311b69") : secret "dns-default-metrics-tls" not found
Nov 01 05:03:00.033544 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:00.033516    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b0122f6-d51a-4338-a631-056ec9311b69-tmp-dir\") pod \"dns-default-g89wb\" (UID: \"0b0122f6-d51a-4338-a631-056ec9311b69\") " pod="openshift-dns/dns-default-g89wb"
Nov 01 05:03:00.033800 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:00.033781    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0b0122f6-d51a-4338-a631-056ec9311b69-config-volume\") pod \"dns-default-g89wb\" (UID: \"0b0122f6-d51a-4338-a631-056ec9311b69\") " pod="openshift-dns/dns-default-g89wb"
Nov 01 05:03:00.037066 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:00.037043    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/af0da37a-a117-49a6-a06e-6cab0e56a40c-cert\") pod \"ingress-canary-ljs77\" (UID: \"af0da37a-a117-49a6-a06e-6cab0e56a40c\") " pod="openshift-ingress-canary/ingress-canary-ljs77"
Nov 01 05:03:00.044986 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:00.044958    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:00.104951 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:00.104888    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd22d\" (UniqueName: \"kubernetes.io/projected/af0da37a-a117-49a6-a06e-6cab0e56a40c-kube-api-access-hd22d\") pod \"ingress-canary-ljs77\" (UID: \"af0da37a-a117-49a6-a06e-6cab0e56a40c\") " pod="openshift-ingress-canary/ingress-canary-ljs77"
Nov 01 05:03:00.106656 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:00.106632    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9njs4\" (UniqueName: \"kubernetes.io/projected/0b0122f6-d51a-4338-a631-056ec9311b69-kube-api-access-9njs4\") pod \"dns-default-g89wb\" (UID: \"0b0122f6-d51a-4338-a631-056ec9311b69\") " pod="openshift-dns/dns-default-g89wb"
Nov 01 05:03:00.116512 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:00.116494    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ljs77"
Nov 01 05:03:00.233567 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:00.233534    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:00.241288 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:00.241263    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-jn2tf"]
Nov 01 05:03:00.276085 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:00.276061    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jn2tf"
Nov 01 05:03:00.303136 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:00.303110    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-jn2tf"]
Nov 01 05:03:00.324160 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:00.324137    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-kkng7"]
Nov 01 05:03:00.341517 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:00.341500    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kkng7"
Nov 01 05:03:00.386581 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:00.386551    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kkng7"]
Nov 01 05:03:00.409276 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:00.409256    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-fkkn4\""
Nov 01 05:03:00.409400 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:00.409375    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 01 05:03:00.409486 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:00.409419    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xvz28\""
Nov 01 05:03:00.415346 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:00.415325    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:00.416583 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:00.416566    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 01 05:03:00.416881 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:00.416862    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 01 05:03:00.416961 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:00.416861    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 01 05:03:00.416961 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:00.416925    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 01 05:03:00.416961 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:00.416925    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 01 05:03:00.435208 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:00.435184    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8fba6aca-0a63-4924-89b2-87575d844ce6-crio-socket\") pod \"insights-runtime-extractor-kkng7\" (UID: \"8fba6aca-0a63-4924-89b2-87575d844ce6\") " pod="openshift-insights/insights-runtime-extractor-kkng7"
Nov 01 05:03:00.435303 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:00.435221    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mg57\" (UniqueName: \"kubernetes.io/projected/7cd84f11-44c9-4ec6-83c2-7db17ab9991c-kube-api-access-8mg57\") pod \"migrator-66df5d496b-jn2tf\" (UID: \"7cd84f11-44c9-4ec6-83c2-7db17ab9991c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jn2tf"
Nov 01 05:03:00.435303 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:00.435236    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8fba6aca-0a63-4924-89b2-87575d844ce6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kkng7\" (UID: \"8fba6aca-0a63-4924-89b2-87575d844ce6\") " pod="openshift-insights/insights-runtime-extractor-kkng7"
Nov 01 05:03:00.435407 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:00.435313    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6nj8\" (UniqueName: \"kubernetes.io/projected/8fba6aca-0a63-4924-89b2-87575d844ce6-kube-api-access-s6nj8\") pod \"insights-runtime-extractor-kkng7\" (UID: \"8fba6aca-0a63-4924-89b2-87575d844ce6\") " pod="openshift-insights/insights-runtime-extractor-kkng7"
Nov 01 05:03:00.435407 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:00.435388    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8fba6aca-0a63-4924-89b2-87575d844ce6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kkng7\" (UID: \"8fba6aca-0a63-4924-89b2-87575d844ce6\") " pod="openshift-insights/insights-runtime-extractor-kkng7"
Nov 01 05:03:00.435517 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:00.435438    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8fba6aca-0a63-4924-89b2-87575d844ce6-data-volume\") pod \"insights-runtime-extractor-kkng7\" (UID: \"8fba6aca-0a63-4924-89b2-87575d844ce6\") " pod="openshift-insights/insights-runtime-extractor-kkng7"
Nov 01 05:03:00.535835 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:00.535776    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ljs77"]
Nov 01 05:03:00.535945 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:00.535929    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8fba6aca-0a63-4924-89b2-87575d844ce6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kkng7\" (UID: \"8fba6aca-0a63-4924-89b2-87575d844ce6\") " pod="openshift-insights/insights-runtime-extractor-kkng7"
Nov 01 05:03:00.535997 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:00.535963    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6nj8\" (UniqueName: \"kubernetes.io/projected/8fba6aca-0a63-4924-89b2-87575d844ce6-kube-api-access-s6nj8\") pod \"insights-runtime-extractor-kkng7\" (UID: \"8fba6aca-0a63-4924-89b2-87575d844ce6\") " pod="openshift-insights/insights-runtime-extractor-kkng7"
Nov 01 05:03:00.535997 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:00.535984    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8fba6aca-0a63-4924-89b2-87575d844ce6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kkng7\" (UID: \"8fba6aca-0a63-4924-89b2-87575d844ce6\") " pod="openshift-insights/insights-runtime-extractor-kkng7"
Nov 01 05:03:00.536082 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:03:00.536053    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 01 05:03:00.536082 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:03:00.536066    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 05:03:00.536161 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:00.536093    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8fba6aca-0a63-4924-89b2-87575d844ce6-data-volume\") pod \"insights-runtime-extractor-kkng7\" (UID: \"8fba6aca-0a63-4924-89b2-87575d844ce6\") " pod="openshift-insights/insights-runtime-extractor-kkng7"
Nov 01 05:03:00.536161 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:03:00.536113    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8fba6aca-0a63-4924-89b2-87575d844ce6-kube-rbac-proxy-cm podName:8fba6aca-0a63-4924-89b2-87575d844ce6 nodeName:}" failed. No retries permitted until 2026-11-01 05:03:01.036095658 +0000 UTC m=+33.236993004 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8fba6aca-0a63-4924-89b2-87575d844ce6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kkng7" (UID: "8fba6aca-0a63-4924-89b2-87575d844ce6") : configmap "kube-rbac-proxy" not found
Nov 01 05:03:00.536161 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:03:00.536149    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fba6aca-0a63-4924-89b2-87575d844ce6-insights-runtime-extractor-tls podName:8fba6aca-0a63-4924-89b2-87575d844ce6 nodeName:}" failed. No retries permitted until 2026-11-01 05:03:01.03613191 +0000 UTC m=+33.237029241 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8fba6aca-0a63-4924-89b2-87575d844ce6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kkng7" (UID: "8fba6aca-0a63-4924-89b2-87575d844ce6") : secret "insights-runtime-extractor-tls" not found
Nov 01 05:03:00.536311 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:00.536227    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0b0122f6-d51a-4338-a631-056ec9311b69-metrics-tls\") pod \"dns-default-g89wb\" (UID: \"0b0122f6-d51a-4338-a631-056ec9311b69\") " pod="openshift-dns/dns-default-g89wb"
Nov 01 05:03:00.536311 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:00.536253    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8fba6aca-0a63-4924-89b2-87575d844ce6-crio-socket\") pod \"insights-runtime-extractor-kkng7\" (UID: \"8fba6aca-0a63-4924-89b2-87575d844ce6\") " pod="openshift-insights/insights-runtime-extractor-kkng7"
Nov 01 05:03:00.536311 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:00.536288    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mg57\" (UniqueName: \"kubernetes.io/projected/7cd84f11-44c9-4ec6-83c2-7db17ab9991c-kube-api-access-8mg57\") pod \"migrator-66df5d496b-jn2tf\" (UID: \"7cd84f11-44c9-4ec6-83c2-7db17ab9991c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jn2tf"
Nov 01 05:03:00.536439 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:03:00.536333    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 05:03:00.536439 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:03:00.536379    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b0122f6-d51a-4338-a631-056ec9311b69-metrics-tls podName:0b0122f6-d51a-4338-a631-056ec9311b69 nodeName:}" failed. No retries permitted until 2026-11-01 05:03:01.536365879 +0000 UTC m=+33.737263208 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0b0122f6-d51a-4338-a631-056ec9311b69-metrics-tls") pod "dns-default-g89wb" (UID: "0b0122f6-d51a-4338-a631-056ec9311b69") : secret "dns-default-metrics-tls" not found
Nov 01 05:03:00.536540 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:00.536447    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8fba6aca-0a63-4924-89b2-87575d844ce6-data-volume\") pod \"insights-runtime-extractor-kkng7\" (UID: \"8fba6aca-0a63-4924-89b2-87575d844ce6\") " pod="openshift-insights/insights-runtime-extractor-kkng7"
Nov 01 05:03:00.536540 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:00.536476    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8fba6aca-0a63-4924-89b2-87575d844ce6-crio-socket\") pod \"insights-runtime-extractor-kkng7\" (UID: \"8fba6aca-0a63-4924-89b2-87575d844ce6\") " pod="openshift-insights/insights-runtime-extractor-kkng7"
Nov 01 05:03:00.539027 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:03:00.539009    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf0da37a_a117_49a6_a06e_6cab0e56a40c.slice/crio-efed176f934c493edca2ddc32acd11fa4382cd5265b4f1516ff814fbb11a2114 WatchSource:0}: Error finding container efed176f934c493edca2ddc32acd11fa4382cd5265b4f1516ff814fbb11a2114: Status 404 returned error can't find the container with id efed176f934c493edca2ddc32acd11fa4382cd5265b4f1516ff814fbb11a2114
Nov 01 05:03:00.551633 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:00.551611    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-t7btw-l6t25"]
Nov 01 05:03:00.584290 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:00.584269    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-t7btw-l6t25"
Nov 01 05:03:00.587797 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:00.587773    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:00.600984 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:00.600963    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6nj8\" (UniqueName: \"kubernetes.io/projected/8fba6aca-0a63-4924-89b2-87575d844ce6-kube-api-access-s6nj8\") pod \"insights-runtime-extractor-kkng7\" (UID: \"8fba6aca-0a63-4924-89b2-87575d844ce6\") " pod="openshift-insights/insights-runtime-extractor-kkng7"
Nov 01 05:03:00.601479 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:00.601464    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mg57\" (UniqueName: \"kubernetes.io/projected/7cd84f11-44c9-4ec6-83c2-7db17ab9991c-kube-api-access-8mg57\") pod \"migrator-66df5d496b-jn2tf\" (UID: \"7cd84f11-44c9-4ec6-83c2-7db17ab9991c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jn2tf"
Nov 01 05:03:00.612854 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:00.612834    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-t7btw-l6t25"]
Nov 01 05:03:00.638832 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:00.638800    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 01 05:03:00.638950 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:00.638876    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-slwj6\""
Nov 01 05:03:00.690611 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:00.690592    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 01 05:03:00.737547 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:00.737526    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f4aae57-6429-4f9e-aaa5-c700bc26b4f5-trusted-ca-bundle\") pod \"periodic-gathering-t7btw-l6t25\" (UID: \"6f4aae57-6429-4f9e-aaa5-c700bc26b4f5\") " pod="openshift-insights/periodic-gathering-t7btw-l6t25"
Nov 01 05:03:00.737628 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:00.737579    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f4aae57-6429-4f9e-aaa5-c700bc26b4f5-service-ca-bundle\") pod \"periodic-gathering-t7btw-l6t25\" (UID: \"6f4aae57-6429-4f9e-aaa5-c700bc26b4f5\") " pod="openshift-insights/periodic-gathering-t7btw-l6t25"
Nov 01 05:03:00.737700 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:00.737680    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-248gt\" (UniqueName: \"kubernetes.io/projected/6f4aae57-6429-4f9e-aaa5-c700bc26b4f5-kube-api-access-248gt\") pod \"periodic-gathering-t7btw-l6t25\" (UID: \"6f4aae57-6429-4f9e-aaa5-c700bc26b4f5\") " pod="openshift-insights/periodic-gathering-t7btw-l6t25"
Nov 01 05:03:00.737737 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:00.737718    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/6f4aae57-6429-4f9e-aaa5-c700bc26b4f5-archives-path\") pod \"periodic-gathering-t7btw-l6t25\" (UID: \"6f4aae57-6429-4f9e-aaa5-c700bc26b4f5\") " pod="openshift-insights/periodic-gathering-t7btw-l6t25"
Nov 01 05:03:00.771376 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:00.771352    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:00.838196 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:00.838141    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f4aae57-6429-4f9e-aaa5-c700bc26b4f5-trusted-ca-bundle\") pod \"periodic-gathering-t7btw-l6t25\" (UID: \"6f4aae57-6429-4f9e-aaa5-c700bc26b4f5\") " pod="openshift-insights/periodic-gathering-t7btw-l6t25"
Nov 01 05:03:00.838196 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:00.838183    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f4aae57-6429-4f9e-aaa5-c700bc26b4f5-service-ca-bundle\") pod \"periodic-gathering-t7btw-l6t25\" (UID: \"6f4aae57-6429-4f9e-aaa5-c700bc26b4f5\") " pod="openshift-insights/periodic-gathering-t7btw-l6t25"
Nov 01 05:03:00.838345 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:00.838225    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-248gt\" (UniqueName: \"kubernetes.io/projected/6f4aae57-6429-4f9e-aaa5-c700bc26b4f5-kube-api-access-248gt\") pod \"periodic-gathering-t7btw-l6t25\" (UID: \"6f4aae57-6429-4f9e-aaa5-c700bc26b4f5\") " pod="openshift-insights/periodic-gathering-t7btw-l6t25"
Nov 01 05:03:00.838345 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:00.838257    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/6f4aae57-6429-4f9e-aaa5-c700bc26b4f5-archives-path\") pod \"periodic-gathering-t7btw-l6t25\" (UID: \"6f4aae57-6429-4f9e-aaa5-c700bc26b4f5\") " pod="openshift-insights/periodic-gathering-t7btw-l6t25"
Nov 01 05:03:00.838669 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:00.838651    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/6f4aae57-6429-4f9e-aaa5-c700bc26b4f5-archives-path\") pod \"periodic-gathering-t7btw-l6t25\" (UID: \"6f4aae57-6429-4f9e-aaa5-c700bc26b4f5\") " pod="openshift-insights/periodic-gathering-t7btw-l6t25"
Nov 01 05:03:00.838831 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:00.838811    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f4aae57-6429-4f9e-aaa5-c700bc26b4f5-service-ca-bundle\") pod \"periodic-gathering-t7btw-l6t25\" (UID: \"6f4aae57-6429-4f9e-aaa5-c700bc26b4f5\") " pod="openshift-insights/periodic-gathering-t7btw-l6t25"
Nov 01 05:03:00.838988 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:00.838973    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f4aae57-6429-4f9e-aaa5-c700bc26b4f5-trusted-ca-bundle\") pod \"periodic-gathering-t7btw-l6t25\" (UID: \"6f4aae57-6429-4f9e-aaa5-c700bc26b4f5\") " pod="openshift-insights/periodic-gathering-t7btw-l6t25"
Nov 01 05:03:00.899508 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:00.899492    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-248gt\" (UniqueName: \"kubernetes.io/projected/6f4aae57-6429-4f9e-aaa5-c700bc26b4f5-kube-api-access-248gt\") pod \"periodic-gathering-t7btw-l6t25\" (UID: \"6f4aae57-6429-4f9e-aaa5-c700bc26b4f5\") " pod="openshift-insights/periodic-gathering-t7btw-l6t25"
Nov 01 05:03:00.900245 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:00.900233    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jn2tf"
Nov 01 05:03:00.915381 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:00.915362    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-t7btw-l6t25"
Nov 01 05:03:00.959757 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:00.959709    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:01.039125 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:01.039097    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8fba6aca-0a63-4924-89b2-87575d844ce6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kkng7\" (UID: \"8fba6aca-0a63-4924-89b2-87575d844ce6\") " pod="openshift-insights/insights-runtime-extractor-kkng7"
Nov 01 05:03:01.039256 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:01.039149    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8fba6aca-0a63-4924-89b2-87575d844ce6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kkng7\" (UID: \"8fba6aca-0a63-4924-89b2-87575d844ce6\") " pod="openshift-insights/insights-runtime-extractor-kkng7"
Nov 01 05:03:01.039321 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:03:01.039269    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 05:03:01.039321 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:03:01.039306    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 01 05:03:01.039412 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:03:01.039356    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fba6aca-0a63-4924-89b2-87575d844ce6-insights-runtime-extractor-tls podName:8fba6aca-0a63-4924-89b2-87575d844ce6 nodeName:}" failed. No retries permitted until 2026-11-01 05:03:02.039332921 +0000 UTC m=+34.240230274 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8fba6aca-0a63-4924-89b2-87575d844ce6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kkng7" (UID: "8fba6aca-0a63-4924-89b2-87575d844ce6") : secret "insights-runtime-extractor-tls" not found
Nov 01 05:03:01.039412 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:03:01.039378    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8fba6aca-0a63-4924-89b2-87575d844ce6-kube-rbac-proxy-cm podName:8fba6aca-0a63-4924-89b2-87575d844ce6 nodeName:}" failed. No retries permitted until 2026-11-01 05:03:02.039366076 +0000 UTC m=+34.240263410 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8fba6aca-0a63-4924-89b2-87575d844ce6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kkng7" (UID: "8fba6aca-0a63-4924-89b2-87575d844ce6") : configmap "kube-rbac-proxy" not found
Nov 01 05:03:01.241106 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:01.241079    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e856df5a-5c5e-4421-ae7d-80a760bc72e2-original-pull-secret\") pod \"global-pull-secret-syncer-9pcxf\" (UID: \"e856df5a-5c5e-4421-ae7d-80a760bc72e2\") " pod="kube-system/global-pull-secret-syncer-9pcxf"
Nov 01 05:03:01.241259 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:01.241139    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4321e0ae-e339-40f2-875a-941b1183e766-metrics-certs\") pod \"network-metrics-daemon-267rv\" (UID: \"4321e0ae-e339-40f2-875a-941b1183e766\") " pod="openshift-multus/network-metrics-daemon-267rv"
Nov 01 05:03:01.241259 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:03:01.241228    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 05:03:01.241345 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:03:01.241290    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e856df5a-5c5e-4421-ae7d-80a760bc72e2-original-pull-secret podName:e856df5a-5c5e-4421-ae7d-80a760bc72e2 nodeName:}" failed. No retries permitted until 2026-11-01 05:03:33.241267765 +0000 UTC m=+65.442165116 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e856df5a-5c5e-4421-ae7d-80a760bc72e2-original-pull-secret") pod "global-pull-secret-syncer-9pcxf" (UID: "e856df5a-5c5e-4421-ae7d-80a760bc72e2") : object "kube-system"/"original-pull-secret" not registered
Nov 01 05:03:01.241345 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:03:01.241232    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 05:03:01.241409 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:03:01.241370    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4321e0ae-e339-40f2-875a-941b1183e766-metrics-certs podName:4321e0ae-e339-40f2-875a-941b1183e766 nodeName:}" failed. No retries permitted until 2026-11-01 05:03:33.24135603 +0000 UTC m=+65.442253361 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4321e0ae-e339-40f2-875a-941b1183e766-metrics-certs") pod "network-metrics-daemon-267rv" (UID: "4321e0ae-e339-40f2-875a-941b1183e766") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 05:03:01.283312 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:01.283288    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-jn2tf"]
Nov 01 05:03:01.286507 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:03:01.286480    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cd84f11_44c9_4ec6_83c2_7db17ab9991c.slice/crio-d69ee7d9a731d0de5d92c0c9c962a0915b8fe7820c45d13c2b059af4a12e65ea WatchSource:0}: Error finding container d69ee7d9a731d0de5d92c0c9c962a0915b8fe7820c45d13c2b059af4a12e65ea: Status 404 returned error can't find the container with id d69ee7d9a731d0de5d92c0c9c962a0915b8fe7820c45d13c2b059af4a12e65ea
Nov 01 05:03:01.307723 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:01.307700    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-267rv"
Nov 01 05:03:01.307825 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:01.307736    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sr4n8"
Nov 01 05:03:01.307872 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:01.307837    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9pcxf"
Nov 01 05:03:01.334211 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:01.334189    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-t7btw-l6t25"]
Nov 01 05:03:01.337290 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:03:01.337269    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f4aae57_6429_4f9e_aaa5_c700bc26b4f5.slice/crio-d1f3f2d5ced62268ee33102b7bc84b719b90c807b6ef49c26c5eec98182f2d13 WatchSource:0}: Error finding container d1f3f2d5ced62268ee33102b7bc84b719b90c807b6ef49c26c5eec98182f2d13: Status 404 returned error can't find the container with id d1f3f2d5ced62268ee33102b7bc84b719b90c807b6ef49c26c5eec98182f2d13
Nov 01 05:03:01.342167 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:01.342147    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffjr4\" (UniqueName: \"kubernetes.io/projected/bcc64278-da5e-4525-90bf-abddde870d70-kube-api-access-ffjr4\") pod \"network-check-target-sr4n8\" (UID: \"bcc64278-da5e-4525-90bf-abddde870d70\") " pod="openshift-network-diagnostics/network-check-target-sr4n8"
Nov 01 05:03:01.361083 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:01.361061    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 01 05:03:01.361160 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:01.361107    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 01 05:03:01.361160 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:01.361152    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 01 05:03:01.361237 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:01.361225    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-7kc5j\""
Nov 01 05:03:01.361291 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:01.361281    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6k7br\""
Nov 01 05:03:01.361320 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:01.361306    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 01 05:03:01.361320 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:01.361303    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-g7ktw\""
Nov 01 05:03:01.364667 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:01.364648    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffjr4\" (UniqueName: \"kubernetes.io/projected/bcc64278-da5e-4525-90bf-abddde870d70-kube-api-access-ffjr4\") pod \"network-check-target-sr4n8\" (UID: \"bcc64278-da5e-4525-90bf-abddde870d70\") " pod="openshift-network-diagnostics/network-check-target-sr4n8"
Nov 01 05:03:01.375180 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:01.375160    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sr4n8"
Nov 01 05:03:01.517630 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:01.517561    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ljs77" event={"ID":"af0da37a-a117-49a6-a06e-6cab0e56a40c","Type":"ContainerStarted","Data":"efed176f934c493edca2ddc32acd11fa4382cd5265b4f1516ff814fbb11a2114"}
Nov 01 05:03:01.518801 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:01.518767    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-t7btw-l6t25" event={"ID":"6f4aae57-6429-4f9e-aaa5-c700bc26b4f5","Type":"ContainerStarted","Data":"d1f3f2d5ced62268ee33102b7bc84b719b90c807b6ef49c26c5eec98182f2d13"}
Nov 01 05:03:01.520408 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:01.520380    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jn2tf" event={"ID":"7cd84f11-44c9-4ec6-83c2-7db17ab9991c","Type":"ContainerStarted","Data":"d69ee7d9a731d0de5d92c0c9c962a0915b8fe7820c45d13c2b059af4a12e65ea"}
Nov 01 05:03:01.543566 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:01.543536    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0b0122f6-d51a-4338-a631-056ec9311b69-metrics-tls\") pod \"dns-default-g89wb\" (UID: \"0b0122f6-d51a-4338-a631-056ec9311b69\") " pod="openshift-dns/dns-default-g89wb"
Nov 01 05:03:01.543718 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:03:01.543700    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 05:03:01.543768 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:03:01.543760    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b0122f6-d51a-4338-a631-056ec9311b69-metrics-tls podName:0b0122f6-d51a-4338-a631-056ec9311b69 nodeName:}" failed. No retries permitted until 2026-11-01 05:03:03.543746142 +0000 UTC m=+35.744643476 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0b0122f6-d51a-4338-a631-056ec9311b69-metrics-tls") pod "dns-default-g89wb" (UID: "0b0122f6-d51a-4338-a631-056ec9311b69") : secret "dns-default-metrics-tls" not found
Nov 01 05:03:01.685906 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:01.685875    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-pwtc2"]
Nov 01 05:03:01.700049 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:01.700019    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-pwtc2"
Nov 01 05:03:01.746657 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:01.746631    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-pwtc2"]
Nov 01 05:03:01.753367 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:01.753330    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Nov 01 05:03:01.753519 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:01.753411    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Nov 01 05:03:01.753590 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:01.753535    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Nov 01 05:03:01.753657 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:01.753639    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Nov 01 05:03:01.753693 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:01.753639    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-bp8rt\""
Nov 01 05:03:01.763675 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:01.763655    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-sr4n8"]
Nov 01 05:03:01.766980 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:03:01.766954    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcc64278_da5e_4525_90bf_abddde870d70.slice/crio-77d30d727f12a09cd823c280b3f077ffd3a480b47977d9f32f3b0ab9be8b4cb5 WatchSource:0}: Error finding container 77d30d727f12a09cd823c280b3f077ffd3a480b47977d9f32f3b0ab9be8b4cb5: Status 404 returned error can't find the container with id 77d30d727f12a09cd823c280b3f077ffd3a480b47977d9f32f3b0ab9be8b4cb5
Nov 01 05:03:01.846595 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:01.846567    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/46078cac-ae78-4ac7-bda3-b95913a478f8-signing-key\") pod \"service-ca-5fffc97bdf-pwtc2\" (UID: \"46078cac-ae78-4ac7-bda3-b95913a478f8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-pwtc2"
Nov 01 05:03:01.846742 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:01.846607    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/46078cac-ae78-4ac7-bda3-b95913a478f8-signing-cabundle\") pod \"service-ca-5fffc97bdf-pwtc2\" (UID: \"46078cac-ae78-4ac7-bda3-b95913a478f8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-pwtc2"
Nov 01 05:03:01.846742 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:01.846727    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djmg6\" (UniqueName: \"kubernetes.io/projected/46078cac-ae78-4ac7-bda3-b95913a478f8-kube-api-access-djmg6\") pod \"service-ca-5fffc97bdf-pwtc2\" (UID: \"46078cac-ae78-4ac7-bda3-b95913a478f8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-pwtc2"
Nov 01 05:03:01.944993 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:01.944963    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:01.947821 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:01.947791    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/46078cac-ae78-4ac7-bda3-b95913a478f8-signing-key\") pod \"service-ca-5fffc97bdf-pwtc2\" (UID: \"46078cac-ae78-4ac7-bda3-b95913a478f8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-pwtc2"
Nov 01 05:03:01.947931 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:01.947841    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/46078cac-ae78-4ac7-bda3-b95913a478f8-signing-cabundle\") pod \"service-ca-5fffc97bdf-pwtc2\" (UID: \"46078cac-ae78-4ac7-bda3-b95913a478f8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-pwtc2"
Nov 01 05:03:01.947931 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:01.947910    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djmg6\" (UniqueName: \"kubernetes.io/projected/46078cac-ae78-4ac7-bda3-b95913a478f8-kube-api-access-djmg6\") pod \"service-ca-5fffc97bdf-pwtc2\" (UID: \"46078cac-ae78-4ac7-bda3-b95913a478f8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-pwtc2"
Nov 01 05:03:01.948640 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:01.948617    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/46078cac-ae78-4ac7-bda3-b95913a478f8-signing-cabundle\") pod \"service-ca-5fffc97bdf-pwtc2\" (UID: \"46078cac-ae78-4ac7-bda3-b95913a478f8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-pwtc2"
Nov 01 05:03:01.950591 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:01.950567    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/46078cac-ae78-4ac7-bda3-b95913a478f8-signing-key\") pod \"service-ca-5fffc97bdf-pwtc2\" (UID: \"46078cac-ae78-4ac7-bda3-b95913a478f8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-pwtc2"
Nov 01 05:03:02.009675 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:02.009654    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djmg6\" (UniqueName: \"kubernetes.io/projected/46078cac-ae78-4ac7-bda3-b95913a478f8-kube-api-access-djmg6\") pod \"service-ca-5fffc97bdf-pwtc2\" (UID: \"46078cac-ae78-4ac7-bda3-b95913a478f8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-pwtc2"
Nov 01 05:03:02.023392 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:02.023349    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-pwtc2"
Nov 01 05:03:02.048904 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:02.048878    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8fba6aca-0a63-4924-89b2-87575d844ce6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kkng7\" (UID: \"8fba6aca-0a63-4924-89b2-87575d844ce6\") " pod="openshift-insights/insights-runtime-extractor-kkng7"
Nov 01 05:03:02.049009 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:02.048933    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8fba6aca-0a63-4924-89b2-87575d844ce6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kkng7\" (UID: \"8fba6aca-0a63-4924-89b2-87575d844ce6\") " pod="openshift-insights/insights-runtime-extractor-kkng7"
Nov 01 05:03:02.049070 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:03:02.049046    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 05:03:02.049070 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:03:02.049062    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 01 05:03:02.049162 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:03:02.049121    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fba6aca-0a63-4924-89b2-87575d844ce6-insights-runtime-extractor-tls podName:8fba6aca-0a63-4924-89b2-87575d844ce6 nodeName:}" failed. No retries permitted until 2026-11-01 05:03:04.049101484 +0000 UTC m=+36.249998818 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8fba6aca-0a63-4924-89b2-87575d844ce6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kkng7" (UID: "8fba6aca-0a63-4924-89b2-87575d844ce6") : secret "insights-runtime-extractor-tls" not found
Nov 01 05:03:02.049162 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:03:02.049145    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8fba6aca-0a63-4924-89b2-87575d844ce6-kube-rbac-proxy-cm podName:8fba6aca-0a63-4924-89b2-87575d844ce6 nodeName:}" failed. No retries permitted until 2026-11-01 05:03:04.049132521 +0000 UTC m=+36.250029856 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8fba6aca-0a63-4924-89b2-87575d844ce6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kkng7" (UID: "8fba6aca-0a63-4924-89b2-87575d844ce6") : configmap "kube-rbac-proxy" not found
Nov 01 05:03:02.167728 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:02.167689    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:02.414644 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:02.414613    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-pwtc2"]
Nov 01 05:03:02.526529 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:02.526494    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-pwtc2" event={"ID":"46078cac-ae78-4ac7-bda3-b95913a478f8","Type":"ContainerStarted","Data":"0711f3d4e1d586814ad4969dd3f79f2539b26f54e5db605f7723168fd44db9d8"}
Nov 01 05:03:02.528215 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:02.528179    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-sr4n8" event={"ID":"bcc64278-da5e-4525-90bf-abddde870d70","Type":"ContainerStarted","Data":"77d30d727f12a09cd823c280b3f077ffd3a480b47977d9f32f3b0ab9be8b4cb5"}
Nov 01 05:03:02.694342 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:02.694271    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:03.561864 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:03.561653    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0b0122f6-d51a-4338-a631-056ec9311b69-metrics-tls\") pod \"dns-default-g89wb\" (UID: \"0b0122f6-d51a-4338-a631-056ec9311b69\") " pod="openshift-dns/dns-default-g89wb"
Nov 01 05:03:03.561864 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:03:03.561803    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 05:03:03.561864 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:03:03.561844    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b0122f6-d51a-4338-a631-056ec9311b69-metrics-tls podName:0b0122f6-d51a-4338-a631-056ec9311b69 nodeName:}" failed. No retries permitted until 2026-11-01 05:03:07.561831728 +0000 UTC m=+39.762729060 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0b0122f6-d51a-4338-a631-056ec9311b69-metrics-tls") pod "dns-default-g89wb" (UID: "0b0122f6-d51a-4338-a631-056ec9311b69") : secret "dns-default-metrics-tls" not found
Nov 01 05:03:04.065687 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:04.065650    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8fba6aca-0a63-4924-89b2-87575d844ce6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kkng7\" (UID: \"8fba6aca-0a63-4924-89b2-87575d844ce6\") " pod="openshift-insights/insights-runtime-extractor-kkng7"
Nov 01 05:03:04.065865 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:04.065717    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8fba6aca-0a63-4924-89b2-87575d844ce6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kkng7\" (UID: \"8fba6aca-0a63-4924-89b2-87575d844ce6\") " pod="openshift-insights/insights-runtime-extractor-kkng7"
Nov 01 05:03:04.065865 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:03:04.065816    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 05:03:04.065865 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:03:04.065824    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 01 05:03:04.066024 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:03:04.065885    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8fba6aca-0a63-4924-89b2-87575d844ce6-kube-rbac-proxy-cm podName:8fba6aca-0a63-4924-89b2-87575d844ce6 nodeName:}" failed. No retries permitted until 2026-11-01 05:03:08.065866674 +0000 UTC m=+40.266764010 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8fba6aca-0a63-4924-89b2-87575d844ce6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kkng7" (UID: "8fba6aca-0a63-4924-89b2-87575d844ce6") : configmap "kube-rbac-proxy" not found
Nov 01 05:03:04.066024 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:03:04.065903    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fba6aca-0a63-4924-89b2-87575d844ce6-insights-runtime-extractor-tls podName:8fba6aca-0a63-4924-89b2-87575d844ce6 nodeName:}" failed. No retries permitted until 2026-11-01 05:03:08.065893432 +0000 UTC m=+40.266790777 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8fba6aca-0a63-4924-89b2-87575d844ce6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kkng7" (UID: "8fba6aca-0a63-4924-89b2-87575d844ce6") : secret "insights-runtime-extractor-tls" not found
Nov 01 05:03:04.537273 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:04.537242    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jn2tf" event={"ID":"7cd84f11-44c9-4ec6-83c2-7db17ab9991c","Type":"ContainerStarted","Data":"43897259244cc39f3ffa5d74f1cf8f7b10f37dfcb8536436ba9b463fc19f4ce2"}
Nov 01 05:03:04.538987 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:04.538935    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ljs77" event={"ID":"af0da37a-a117-49a6-a06e-6cab0e56a40c","Type":"ContainerStarted","Data":"e7d1614b20b8f0718ba94591c7e4eee8ebbb1c9249d25d904263334cec484910"}
Nov 01 05:03:04.655365 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:04.654667    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jn2tf" podStartSLOduration=2.537492467 podStartE2EDuration="4.654647099s" totalImagesPullingTime="2.117154632s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:03:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:03:04.654494007 +0000 UTC m=+36.855391378" watchObservedRunningTime="2026-11-01 05:03:04.654647099 +0000 UTC m=+36.855544454"
Nov 01 05:03:04.762842 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:04.762748    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:05.356080 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:05.356043    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:05.543214 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:05.543175    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-t7btw-l6t25" event={"ID":"6f4aae57-6429-4f9e-aaa5-c700bc26b4f5","Type":"ContainerStarted","Data":"926cf2e9417e15c25ee90ffb91fa457785273e7901c6f240cbc6763bc47826a8"}
Nov 01 05:03:05.661746 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:05.661022    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-t7btw-l6t25" podStartSLOduration=2.222940457 podStartE2EDuration="5.661005588s" totalImagesPullingTime="3.438065131s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:03:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:03:05.660697533 +0000 UTC m=+37.861594884" watchObservedRunningTime="2026-11-01 05:03:05.661005588 +0000 UTC m=+37.861902938"
Nov 01 05:03:05.782497 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:05.781841    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-ljs77" podStartSLOduration=2.864123178 podStartE2EDuration="6.781826622s" totalImagesPullingTime="3.917703444s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:02:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:03:05.781112172 +0000 UTC m=+37.982009526" watchObservedRunningTime="2026-11-01 05:03:05.781826622 +0000 UTC m=+37.982723972"
Nov 01 05:03:06.546931 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:06.546843    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-pwtc2" event={"ID":"46078cac-ae78-4ac7-bda3-b95913a478f8","Type":"ContainerStarted","Data":"679dc2beecfe9cfb6140e416fefa020cc5b3efacb99b55d5de049b76133caa03"}
Nov 01 05:03:06.548424 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:06.548382    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-sr4n8" event={"ID":"bcc64278-da5e-4525-90bf-abddde870d70","Type":"ContainerStarted","Data":"f1befccf96f9a201ee08bba9b4fc99ca2afbf7e7c73b67c27e31177663677ca3"}
Nov 01 05:03:06.663637 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:06.662964    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-pwtc2" podStartSLOduration=2.297259558 podStartE2EDuration="5.662950527s" totalImagesPullingTime="3.365690969s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:03:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:03:06.66257982 +0000 UTC m=+38.863477171" watchObservedRunningTime="2026-11-01 05:03:06.662950527 +0000 UTC m=+38.863847908"
Nov 01 05:03:06.778909 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:06.778860    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-sr4n8" podStartSLOduration=33.327917689 podStartE2EDuration="37.778847415s" totalImagesPullingTime="4.450929726s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:02:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:03:06.777937069 +0000 UTC m=+38.978834422" watchObservedRunningTime="2026-11-01 05:03:06.778847415 +0000 UTC m=+38.979744766"
Nov 01 05:03:07.018792 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:07.018755    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:07.316341 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:07.316255    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:07.496556 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:07.496526    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:07.552521 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:07.552494    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-sr4n8"
Nov 01 05:03:07.597025 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:07.596953    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0b0122f6-d51a-4338-a631-056ec9311b69-metrics-tls\") pod \"dns-default-g89wb\" (UID: \"0b0122f6-d51a-4338-a631-056ec9311b69\") " pod="openshift-dns/dns-default-g89wb"
Nov 01 05:03:07.597138 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:03:07.597114    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 05:03:07.597192 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:03:07.597181    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b0122f6-d51a-4338-a631-056ec9311b69-metrics-tls podName:0b0122f6-d51a-4338-a631-056ec9311b69 nodeName:}" failed. No retries permitted until 2026-11-01 05:03:15.597162601 +0000 UTC m=+47.798059938 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0b0122f6-d51a-4338-a631-056ec9311b69-metrics-tls") pod "dns-default-g89wb" (UID: "0b0122f6-d51a-4338-a631-056ec9311b69") : secret "dns-default-metrics-tls" not found
Nov 01 05:03:08.061234 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:08.061192    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:08.101066 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:08.101032    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8fba6aca-0a63-4924-89b2-87575d844ce6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kkng7\" (UID: \"8fba6aca-0a63-4924-89b2-87575d844ce6\") " pod="openshift-insights/insights-runtime-extractor-kkng7"
Nov 01 05:03:08.101199 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:08.101081    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8fba6aca-0a63-4924-89b2-87575d844ce6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kkng7\" (UID: \"8fba6aca-0a63-4924-89b2-87575d844ce6\") " pod="openshift-insights/insights-runtime-extractor-kkng7"
Nov 01 05:03:08.101199 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:03:08.101146    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 05:03:08.101267 ip-10-0-129-92 kubenswrapper[2571]: E1101 05:03:08.101213    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fba6aca-0a63-4924-89b2-87575d844ce6-insights-runtime-extractor-tls podName:8fba6aca-0a63-4924-89b2-87575d844ce6 nodeName:}" failed. No retries permitted until 2026-11-01 05:03:16.101198767 +0000 UTC m=+48.302096111 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8fba6aca-0a63-4924-89b2-87575d844ce6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kkng7" (UID: "8fba6aca-0a63-4924-89b2-87575d844ce6") : secret "insights-runtime-extractor-tls" not found
Nov 01 05:03:08.101552 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:08.101537    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8fba6aca-0a63-4924-89b2-87575d844ce6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kkng7\" (UID: \"8fba6aca-0a63-4924-89b2-87575d844ce6\") " pod="openshift-insights/insights-runtime-extractor-kkng7"
Nov 01 05:03:08.661388 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:08.661351    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:08.839481 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:08.839291    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:09.014907 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:09.014828    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:09.099731 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:09.099707    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ddc5d_a877f895-a4cf-404d-87e4-440d4b73a870/dns-node-resolver/0.log"
Nov 01 05:03:09.196370 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:09.196326    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:09.362993 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:09.362923    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:09.554476 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:09.554406    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:10.057945 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:10.057906    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:10.099159 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:10.099142    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vrbks_d454e879-03a2-4fb9-8223-79a1aa9d85d2/node-ca/0.log"
Nov 01 05:03:10.656235 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:10.656201    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:10.698591 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:10.698568    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ljs77_af0da37a-a117-49a6-a06e-6cab0e56a40c/serve-healthcheck-canary/0.log"
Nov 01 05:03:10.869368 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:10.869333    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:11.054782 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:11.054711    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:11.299960 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:11.299937    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-jn2tf_7cd84f11-44c9-4ec6-83c2-7db17ab9991c/migrator/0.log"
Nov 01 05:03:11.490475 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:11.490427    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:12.257286 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:12.257253    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:12.657615 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:12.657580    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:12.860078 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:12.860048    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:13.069222 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:13.069148    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:13.266616 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:13.266581    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:13.801895 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:13.801857    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:14.461122 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:14.461090    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:14.661417 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:14.661383    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:14.861609 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:14.861500    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:15.065798 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:15.065763    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:15.266002 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:15.265963    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:15.660093 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:15.660061    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0b0122f6-d51a-4338-a631-056ec9311b69-metrics-tls\") pod \"dns-default-g89wb\" (UID: \"0b0122f6-d51a-4338-a631-056ec9311b69\") " pod="openshift-dns/dns-default-g89wb"
Nov 01 05:03:15.662410 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:15.662385    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0b0122f6-d51a-4338-a631-056ec9311b69-metrics-tls\") pod \"dns-default-g89wb\" (UID: \"0b0122f6-d51a-4338-a631-056ec9311b69\") " pod="openshift-dns/dns-default-g89wb"
Nov 01 05:03:15.707591 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:15.707567    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-g89wb"
Nov 01 05:03:16.083244 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:16.082684    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-g89wb"]
Nov 01 05:03:16.084882 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:03:16.084857    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b0122f6_d51a_4338_a631_056ec9311b69.slice/crio-46168ed50cbd4d5ada910d8a21a69f2d8d81f28884def6ccc1f353dd392a4090 WatchSource:0}: Error finding container 46168ed50cbd4d5ada910d8a21a69f2d8d81f28884def6ccc1f353dd392a4090: Status 404 returned error can't find the container with id 46168ed50cbd4d5ada910d8a21a69f2d8d81f28884def6ccc1f353dd392a4090
Nov 01 05:03:16.169184 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:16.169161    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8fba6aca-0a63-4924-89b2-87575d844ce6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kkng7\" (UID: \"8fba6aca-0a63-4924-89b2-87575d844ce6\") " pod="openshift-insights/insights-runtime-extractor-kkng7"
Nov 01 05:03:16.171410 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:16.171388    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8fba6aca-0a63-4924-89b2-87575d844ce6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kkng7\" (UID: \"8fba6aca-0a63-4924-89b2-87575d844ce6\") " pod="openshift-insights/insights-runtime-extractor-kkng7"
Nov 01 05:03:16.288139 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:16.288112    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kkng7"
Nov 01 05:03:16.460573 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:16.460537    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:16.573838 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:16.573799    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-g89wb" event={"ID":"0b0122f6-d51a-4338-a631-056ec9311b69","Type":"ContainerStarted","Data":"46168ed50cbd4d5ada910d8a21a69f2d8d81f28884def6ccc1f353dd392a4090"}
Nov 01 05:03:16.657267 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:16.657234    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kkng7"]
Nov 01 05:03:16.660075 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:03:16.660042    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fba6aca_0a63_4924_89b2_87575d844ce6.slice/crio-4c6b70bec32c1ece062d94681b27d82605927e2b81b18ab88589b471bb233102 WatchSource:0}: Error finding container 4c6b70bec32c1ece062d94681b27d82605927e2b81b18ab88589b471bb233102: Status 404 returned error can't find the container with id 4c6b70bec32c1ece062d94681b27d82605927e2b81b18ab88589b471bb233102
Nov 01 05:03:16.687614 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:16.687590    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:16.865750 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:16.865715    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:17.038405 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:17.038327    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:17.216099 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:17.216060    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:17.385862 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:17.385774    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:17.578073 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:17.578036    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kkng7" event={"ID":"8fba6aca-0a63-4924-89b2-87575d844ce6","Type":"ContainerStarted","Data":"b00e290740fd6332a46d7fc922da2fc252164c852ed061c3f785e4f2a970b1a4"}
Nov 01 05:03:17.578231 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:17.578095    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kkng7" event={"ID":"8fba6aca-0a63-4924-89b2-87575d844ce6","Type":"ContainerStarted","Data":"4c6b70bec32c1ece062d94681b27d82605927e2b81b18ab88589b471bb233102"}
Nov 01 05:03:17.596827 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:17.596798    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:17.775063 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:17.775029    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:17.947703 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:17.947674    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:18.121660 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:18.121632    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:18.463602 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:18.463573    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:18.583145 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:18.583114    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kkng7" event={"ID":"8fba6aca-0a63-4924-89b2-87575d844ce6","Type":"ContainerStarted","Data":"dcf33d35cfd494a415a0461b775c7b9fa2f6291874fe6d63a55d7cb3e2875f67"}
Nov 01 05:03:18.584828 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:18.584796    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-g89wb" event={"ID":"0b0122f6-d51a-4338-a631-056ec9311b69","Type":"ContainerStarted","Data":"d4839441beb0b44367df7bb9b140b363cccac106362bf869f9e075e0bc1f6ecc"}
Nov 01 05:03:18.584944 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:18.584845    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-g89wb" event={"ID":"0b0122f6-d51a-4338-a631-056ec9311b69","Type":"ContainerStarted","Data":"a2883fb922ffea4d8bf09f59937f928830a7a8a1010ba2dae9547350d3d5e880"}
Nov 01 05:03:18.584944 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:18.584936    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-g89wb"
Nov 01 05:03:18.703715 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:18.703663    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-g89wb" podStartSLOduration=17.770384741 podStartE2EDuration="19.703652081s" totalImagesPullingTime="1.93326734s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:02:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:03:18.703118221 +0000 UTC m=+50.904015573" watchObservedRunningTime="2026-11-01 05:03:18.703652081 +0000 UTC m=+50.904549432"
Nov 01 05:03:19.456070 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:19.456020    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:20.593064 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:20.593031    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kkng7" event={"ID":"8fba6aca-0a63-4924-89b2-87575d844ce6","Type":"ContainerStarted","Data":"1bc4700989563e3d9ab0d54cd05e98c3dc3a6b054b038998b23b83740d3ccbcc"}
Nov 01 05:03:20.713127 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:20.713083    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-kkng7" podStartSLOduration=17.841491311 podStartE2EDuration="20.713069719s" totalImagesPullingTime="2.871578408s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 05:03:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:03:20.712591056 +0000 UTC m=+52.913488411" watchObservedRunningTime="2026-11-01 05:03:20.713069719 +0000 UTC m=+52.913967061"
Nov 01 05:03:21.873428 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:21.873397    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-fbbdp"]
Nov 01 05:03:21.875014 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:21.874996    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-645d98cc97-fdph2"]
Nov 01 05:03:21.875143 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:21.875121    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-fbbdp"
Nov 01 05:03:21.877380 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:21.877360    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-645d98cc97-fdph2"
Nov 01 05:03:21.910652 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:21.910625    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgwkh\" (UniqueName: \"kubernetes.io/projected/ca43c15d-cefb-4e78-93aa-a98573da4feb-kube-api-access-wgwkh\") pod \"console-645d98cc97-fdph2\" (UID: \"ca43c15d-cefb-4e78-93aa-a98573da4feb\") " pod="openshift-console/console-645d98cc97-fdph2"
Nov 01 05:03:21.910778 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:21.910665    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/35391db2-ab6e-4272-8580-9014e24f72c4-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-fbbdp\" (UID: \"35391db2-ab6e-4272-8580-9014e24f72c4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-fbbdp"
Nov 01 05:03:21.910842 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:21.910786    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ca43c15d-cefb-4e78-93aa-a98573da4feb-oauth-serving-cert\") pod \"console-645d98cc97-fdph2\" (UID: \"ca43c15d-cefb-4e78-93aa-a98573da4feb\") " pod="openshift-console/console-645d98cc97-fdph2"
Nov 01 05:03:21.910842 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:21.910825    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ca43c15d-cefb-4e78-93aa-a98573da4feb-console-oauth-config\") pod \"console-645d98cc97-fdph2\" (UID: \"ca43c15d-cefb-4e78-93aa-a98573da4feb\") " pod="openshift-console/console-645d98cc97-fdph2"
Nov 01 05:03:21.910924 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:21.910902    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ca43c15d-cefb-4e78-93aa-a98573da4feb-console-config\") pod \"console-645d98cc97-fdph2\" (UID: \"ca43c15d-cefb-4e78-93aa-a98573da4feb\") " pod="openshift-console/console-645d98cc97-fdph2"
Nov 01 05:03:21.910959 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:21.910946    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ca43c15d-cefb-4e78-93aa-a98573da4feb-service-ca\") pod \"console-645d98cc97-fdph2\" (UID: \"ca43c15d-cefb-4e78-93aa-a98573da4feb\") " pod="openshift-console/console-645d98cc97-fdph2"
Nov 01 05:03:21.911008 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:21.910997    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca43c15d-cefb-4e78-93aa-a98573da4feb-tmp\") pod \"console-645d98cc97-fdph2\" (UID: \"ca43c15d-cefb-4e78-93aa-a98573da4feb\") " pod="openshift-console/console-645d98cc97-fdph2"
Nov 01 05:03:21.911040 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:21.911018    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ca43c15d-cefb-4e78-93aa-a98573da4feb-console-serving-cert\") pod \"console-645d98cc97-fdph2\" (UID: \"ca43c15d-cefb-4e78-93aa-a98573da4feb\") " pod="openshift-console/console-645d98cc97-fdph2"
Nov 01 05:03:21.923962 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:21.923941    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-x7wfp"]
Nov 01 05:03:21.925421 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:21.925408    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-x7wfp"
Nov 01 05:03:21.973757 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:21.973732    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Nov 01 05:03:21.973902 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:21.973822    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-sxmdz\""
Nov 01 05:03:21.973983 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:21.973920    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Nov 01 05:03:21.973983 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:21.973936    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Nov 01 05:03:21.974112 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:21.973984    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-zskd9\""
Nov 01 05:03:21.974112 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:21.974021    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Nov 01 05:03:21.974112 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:21.974038    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 01 05:03:21.979899 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:21.979879    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Nov 01 05:03:21.980938 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:21.980924    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-fbbdp"]
Nov 01 05:03:21.981642 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:21.981626    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-645d98cc97-fdph2"]
Nov 01 05:03:21.982535 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:21.982522    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-x7wfp"]
Nov 01 05:03:22.012184 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:22.012159    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ca43c15d-cefb-4e78-93aa-a98573da4feb-console-config\") pod \"console-645d98cc97-fdph2\" (UID: \"ca43c15d-cefb-4e78-93aa-a98573da4feb\") " pod="openshift-console/console-645d98cc97-fdph2"
Nov 01 05:03:22.012260 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:22.012183    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ca43c15d-cefb-4e78-93aa-a98573da4feb-service-ca\") pod \"console-645d98cc97-fdph2\" (UID: \"ca43c15d-cefb-4e78-93aa-a98573da4feb\") " pod="openshift-console/console-645d98cc97-fdph2"
Nov 01 05:03:22.012260 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:22.012205    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zn2l\" (UniqueName: \"kubernetes.io/projected/a8db8643-51bf-4bcf-b26d-bd937a67b5ed-kube-api-access-4zn2l\") pod \"downloads-6dd789c8b7-x7wfp\" (UID: \"a8db8643-51bf-4bcf-b26d-bd937a67b5ed\") " pod="openshift-console/downloads-6dd789c8b7-x7wfp"
Nov 01 05:03:22.012260 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:22.012250    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgwkh\" (UniqueName: \"kubernetes.io/projected/ca43c15d-cefb-4e78-93aa-a98573da4feb-kube-api-access-wgwkh\") pod \"console-645d98cc97-fdph2\" (UID: \"ca43c15d-cefb-4e78-93aa-a98573da4feb\") " pod="openshift-console/console-645d98cc97-fdph2"
Nov 01 05:03:22.012363 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:22.012337    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a8db8643-51bf-4bcf-b26d-bd937a67b5ed-tmp\") pod \"downloads-6dd789c8b7-x7wfp\" (UID: \"a8db8643-51bf-4bcf-b26d-bd937a67b5ed\") " pod="openshift-console/downloads-6dd789c8b7-x7wfp"
Nov 01 05:03:22.012433 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:22.012416    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca43c15d-cefb-4e78-93aa-a98573da4feb-tmp\") pod \"console-645d98cc97-fdph2\" (UID: \"ca43c15d-cefb-4e78-93aa-a98573da4feb\") " pod="openshift-console/console-645d98cc97-fdph2"
Nov 01 05:03:22.012494 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:22.012480    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ca43c15d-cefb-4e78-93aa-a98573da4feb-console-serving-cert\") pod \"console-645d98cc97-fdph2\" (UID: \"ca43c15d-cefb-4e78-93aa-a98573da4feb\") " pod="openshift-console/console-645d98cc97-fdph2"
Nov 01 05:03:22.012549 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:22.012517    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/35391db2-ab6e-4272-8580-9014e24f72c4-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-fbbdp\" (UID: \"35391db2-ab6e-4272-8580-9014e24f72c4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-fbbdp"
Nov 01 05:03:22.012591 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:22.012562    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ca43c15d-cefb-4e78-93aa-a98573da4feb-oauth-serving-cert\") pod \"console-645d98cc97-fdph2\" (UID: \"ca43c15d-cefb-4e78-93aa-a98573da4feb\") " pod="openshift-console/console-645d98cc97-fdph2"
Nov 01 05:03:22.012591 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:22.012584    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ca43c15d-cefb-4e78-93aa-a98573da4feb-console-oauth-config\") pod \"console-645d98cc97-fdph2\" (UID: \"ca43c15d-cefb-4e78-93aa-a98573da4feb\") " pod="openshift-console/console-645d98cc97-fdph2"
Nov 01 05:03:22.012801 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:22.012775    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca43c15d-cefb-4e78-93aa-a98573da4feb-tmp\") pod \"console-645d98cc97-fdph2\" (UID: \"ca43c15d-cefb-4e78-93aa-a98573da4feb\") " pod="openshift-console/console-645d98cc97-fdph2"
Nov 01 05:03:22.013007 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:22.012987    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ca43c15d-cefb-4e78-93aa-a98573da4feb-console-config\") pod \"console-645d98cc97-fdph2\" (UID: \"ca43c15d-cefb-4e78-93aa-a98573da4feb\") " pod="openshift-console/console-645d98cc97-fdph2"
Nov 01 05:03:22.013442 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:22.013412    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ca43c15d-cefb-4e78-93aa-a98573da4feb-oauth-serving-cert\") pod \"console-645d98cc97-fdph2\" (UID: \"ca43c15d-cefb-4e78-93aa-a98573da4feb\") " pod="openshift-console/console-645d98cc97-fdph2"
Nov 01 05:03:22.015039 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:22.015019    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ca43c15d-cefb-4e78-93aa-a98573da4feb-console-serving-cert\") pod \"console-645d98cc97-fdph2\" (UID: \"ca43c15d-cefb-4e78-93aa-a98573da4feb\") " pod="openshift-console/console-645d98cc97-fdph2"
Nov 01 05:03:22.015106 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:22.015019    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/35391db2-ab6e-4272-8580-9014e24f72c4-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-fbbdp\" (UID: \"35391db2-ab6e-4272-8580-9014e24f72c4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-fbbdp"
Nov 01 05:03:22.015106 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:22.015028    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ca43c15d-cefb-4e78-93aa-a98573da4feb-console-oauth-config\") pod \"console-645d98cc97-fdph2\" (UID: \"ca43c15d-cefb-4e78-93aa-a98573da4feb\") " pod="openshift-console/console-645d98cc97-fdph2"
Nov 01 05:03:22.024472 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:22.024430    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 01 05:03:22.024647 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:22.024624    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Nov 01 05:03:22.030891 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:22.030876    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-62qzp\""
Nov 01 05:03:22.033351 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:22.033330    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ca43c15d-cefb-4e78-93aa-a98573da4feb-service-ca\") pod \"console-645d98cc97-fdph2\" (UID: \"ca43c15d-cefb-4e78-93aa-a98573da4feb\") " pod="openshift-console/console-645d98cc97-fdph2"
Nov 01 05:03:22.038344 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:22.038325    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:22.073888 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:22.073868    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgwkh\" (UniqueName: \"kubernetes.io/projected/ca43c15d-cefb-4e78-93aa-a98573da4feb-kube-api-access-wgwkh\") pod \"console-645d98cc97-fdph2\" (UID: \"ca43c15d-cefb-4e78-93aa-a98573da4feb\") " pod="openshift-console/console-645d98cc97-fdph2"
Nov 01 05:03:22.113859 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:22.113532    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zn2l\" (UniqueName: \"kubernetes.io/projected/a8db8643-51bf-4bcf-b26d-bd937a67b5ed-kube-api-access-4zn2l\") pod \"downloads-6dd789c8b7-x7wfp\" (UID: \"a8db8643-51bf-4bcf-b26d-bd937a67b5ed\") " pod="openshift-console/downloads-6dd789c8b7-x7wfp"
Nov 01 05:03:22.113859 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:22.113636    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a8db8643-51bf-4bcf-b26d-bd937a67b5ed-tmp\") pod \"downloads-6dd789c8b7-x7wfp\" (UID: \"a8db8643-51bf-4bcf-b26d-bd937a67b5ed\") " pod="openshift-console/downloads-6dd789c8b7-x7wfp"
Nov 01 05:03:22.115102 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:22.114191    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a8db8643-51bf-4bcf-b26d-bd937a67b5ed-tmp\") pod \"downloads-6dd789c8b7-x7wfp\" (UID: \"a8db8643-51bf-4bcf-b26d-bd937a67b5ed\") " pod="openshift-console/downloads-6dd789c8b7-x7wfp"
Nov 01 05:03:22.183071 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:22.183042    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zn2l\" (UniqueName: \"kubernetes.io/projected/a8db8643-51bf-4bcf-b26d-bd937a67b5ed-kube-api-access-4zn2l\") pod \"downloads-6dd789c8b7-x7wfp\" (UID: \"a8db8643-51bf-4bcf-b26d-bd937a67b5ed\") " pod="openshift-console/downloads-6dd789c8b7-x7wfp"
Nov 01 05:03:22.185888 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:22.185868    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-fbbdp"
Nov 01 05:03:22.191862 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:22.191844    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-645d98cc97-fdph2"
Nov 01 05:03:22.234770 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:22.234749    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-x7wfp"
Nov 01 05:03:22.239699 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:22.239617    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:22.420617 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:22.420576    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:22.561162 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:22.561094    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-fbbdp"]
Nov 01 05:03:22.563831 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:03:22.563809    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35391db2_ab6e_4272_8580_9014e24f72c4.slice/crio-4255f96b8c68d9e9d843b9faaca419295cf827087a981964731ed862bdadda7c WatchSource:0}: Error finding container 4255f96b8c68d9e9d843b9faaca419295cf827087a981964731ed862bdadda7c: Status 404 returned error can't find the container with id 4255f96b8c68d9e9d843b9faaca419295cf827087a981964731ed862bdadda7c
Nov 01 05:03:22.599671 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:22.599633    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:22.599903 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:22.599884    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-fbbdp" event={"ID":"35391db2-ab6e-4272-8580-9014e24f72c4","Type":"ContainerStarted","Data":"4255f96b8c68d9e9d843b9faaca419295cf827087a981964731ed862bdadda7c"}
Nov 01 05:03:22.611949 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:22.611927    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-645d98cc97-fdph2"]
Nov 01 05:03:22.614643 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:03:22.614622    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca43c15d_cefb_4e78_93aa_a98573da4feb.slice/crio-78cfa2ceed0dfad5311f2c8230151b15c354cce14a4b7b45aa09dac43e4efa53 WatchSource:0}: Error finding container 78cfa2ceed0dfad5311f2c8230151b15c354cce14a4b7b45aa09dac43e4efa53: Status 404 returned error can't find the container with id 78cfa2ceed0dfad5311f2c8230151b15c354cce14a4b7b45aa09dac43e4efa53
Nov 01 05:03:22.665094 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:22.665075    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-x7wfp"]
Nov 01 05:03:22.667926 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:03:22.667903    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8db8643_51bf_4bcf_b26d_bd937a67b5ed.slice/crio-5db42fabf816db16908b2a5afc5dbc7358a7c1d9679a9bbe1be0cdc2008a9469 WatchSource:0}: Error finding container 5db42fabf816db16908b2a5afc5dbc7358a7c1d9679a9bbe1be0cdc2008a9469: Status 404 returned error can't find the container with id 5db42fabf816db16908b2a5afc5dbc7358a7c1d9679a9bbe1be0cdc2008a9469
Nov 01 05:03:22.781569 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:22.781543    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:23.510177 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:23.510144    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:03:23.604806 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:23.604773    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-x7wfp" event={"ID":"a8db8643-51bf-4bcf-b26d-bd937a67b5ed","Type":"ContainerStarted","Data":"5db42fabf816db16908b2a5afc5dbc7358a7c1d9679a9bbe1be0cdc2008a9469"}
Nov 01 05:03:23.606240 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:23.606212    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-645d98cc97-fdph2" event={"ID":"ca43c15d-cefb-4e78-93aa-a98573da4feb","Type":"ContainerStarted","Data":"78cfa2ceed0dfad5311f2c8230151b15c354cce14a4b7b45aa09dac43e4efa53"}
Nov 01 05:03:23.748988 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:23.748951    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:23.935325 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:23.935293    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:24.611836 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:24.611794    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-fbbdp" event={"ID":"35391db2-ab6e-4272-8580-9014e24f72c4","Type":"ContainerStarted","Data":"e85eb7550269b32b59078f434570c11be539ab5968c45e9d8232a627a2b77ebc"}
Nov 01 05:03:24.612371 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:24.612313    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-fbbdp"
Nov 01 05:03:24.618021 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:24.617973    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-fbbdp"
Nov 01 05:03:24.728810 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:24.728755    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-fbbdp" podStartSLOduration=2.197451688 podStartE2EDuration="3.728739985s" totalImagesPullingTime="1.531288297s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:03:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:03:24.728226897 +0000 UTC m=+56.929124282" watchObservedRunningTime="2026-11-01 05:03:24.728739985 +0000 UTC m=+56.929637336"
Nov 01 05:03:24.816533 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:24.816501    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:25.045228 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:25.045189    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:25.221318 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:25.221273    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:25.849307 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:25.849272    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:26.072067 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:26.072027    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:26.250192 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:26.250167    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:26.428271 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:26.428222    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:26.622090 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:26.622007    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-645d98cc97-fdph2" event={"ID":"ca43c15d-cefb-4e78-93aa-a98573da4feb","Type":"ContainerStarted","Data":"ab788a84b045eb570edb9df7c5f6bebf57601753563f826de606de4849b6d5f1"}
Nov 01 05:03:26.742665 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:26.742616    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-645d98cc97-fdph2" podStartSLOduration=2.143407307 podStartE2EDuration="5.742602192s" totalImagesPullingTime="3.599194885s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:03:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:03:26.741780563 +0000 UTC m=+58.942677915" watchObservedRunningTime="2026-11-01 05:03:26.742602192 +0000 UTC m=+58.943499543"
Nov 01 05:03:26.745640 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:26.745602    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:26.926924 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:26.926889    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:27.419441 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:27.419397    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:27.592391 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:27.592354    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:27.767876 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:27.767799    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:28.591587 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:28.591556    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-g89wb"
Nov 01 05:03:31.165068 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:31.165034    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:31.417626 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:31.417542    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:31.598976 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:31.598889    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:31.788416 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:31.788339    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:31.889406 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:31.889372    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-vh7rl"]
Nov 01 05:03:31.893835 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:31.893809    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-5hwhj"]
Nov 01 05:03:31.893975 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:31.893959    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vh7rl"
Nov 01 05:03:31.897253 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:31.897234    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5hwhj"
Nov 01 05:03:31.948432 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:31.948400    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 01 05:03:31.948619 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:31.948602    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 01 05:03:31.948698 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:31.948667    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 01 05:03:31.948759 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:31.948709    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 01 05:03:31.948759 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:31.948724    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-t567f\""
Nov 01 05:03:31.948851 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:31.948785    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 01 05:03:31.974391 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:31.974368    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:31.996363 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:31.996328    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/805c87a3-63e9-4c13-9c90-c05f73282edc-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-5hwhj\" (UID: \"805c87a3-63e9-4c13-9c90-c05f73282edc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5hwhj"
Nov 01 05:03:31.996492 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:31.996368    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/805c87a3-63e9-4c13-9c90-c05f73282edc-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-5hwhj\" (UID: \"805c87a3-63e9-4c13-9c90-c05f73282edc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5hwhj"
Nov 01 05:03:31.996492 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:31.996401    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/805c87a3-63e9-4c13-9c90-c05f73282edc-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-5hwhj\" (UID: \"805c87a3-63e9-4c13-9c90-c05f73282edc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5hwhj"
Nov 01 05:03:31.996607 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:31.996528    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5zxr\" (UniqueName: \"kubernetes.io/projected/9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14-kube-api-access-r5zxr\") pod \"node-exporter-vh7rl\" (UID: \"9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14\") " pod="openshift-monitoring/node-exporter-vh7rl"
Nov 01 05:03:31.996607 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:31.996565    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14-node-exporter-accelerators-collector-config\") pod \"node-exporter-vh7rl\" (UID: \"9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14\") " pod="openshift-monitoring/node-exporter-vh7rl"
Nov 01 05:03:31.996706 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:31.996635    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/805c87a3-63e9-4c13-9c90-c05f73282edc-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-5hwhj\" (UID: \"805c87a3-63e9-4c13-9c90-c05f73282edc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5hwhj"
Nov 01 05:03:31.996706 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:31.996661    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14-node-exporter-textfile\") pod \"node-exporter-vh7rl\" (UID: \"9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14\") " pod="openshift-monitoring/node-exporter-vh7rl"
Nov 01 05:03:31.996706 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:31.996680    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14-node-exporter-wtmp\") pod \"node-exporter-vh7rl\" (UID: \"9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14\") " pod="openshift-monitoring/node-exporter-vh7rl"
Nov 01 05:03:31.996706 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:31.996705    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14-sys\") pod \"node-exporter-vh7rl\" (UID: \"9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14\") " pod="openshift-monitoring/node-exporter-vh7rl"
Nov 01 05:03:31.996889 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:31.996847    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14-root\") pod \"node-exporter-vh7rl\" (UID: \"9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14\") " pod="openshift-monitoring/node-exporter-vh7rl"
Nov 01 05:03:31.996936 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:31.996906    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14-node-exporter-tls\") pod \"node-exporter-vh7rl\" (UID: \"9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14\") " pod="openshift-monitoring/node-exporter-vh7rl"
Nov 01 05:03:31.996987 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:31.996932    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/805c87a3-63e9-4c13-9c90-c05f73282edc-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-5hwhj\" (UID: \"805c87a3-63e9-4c13-9c90-c05f73282edc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5hwhj"
Nov 01 05:03:31.996987 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:31.996953    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vh7rl\" (UID: \"9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14\") " pod="openshift-monitoring/node-exporter-vh7rl"
Nov 01 05:03:31.997072 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:31.997026    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvgxl\" (UniqueName: \"kubernetes.io/projected/805c87a3-63e9-4c13-9c90-c05f73282edc-kube-api-access-hvgxl\") pod \"kube-state-metrics-5dc8d7d89b-5hwhj\" (UID: \"805c87a3-63e9-4c13-9c90-c05f73282edc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5hwhj"
Nov 01 05:03:31.997152 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:31.997139    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14-metrics-client-ca\") pod \"node-exporter-vh7rl\" (UID: \"9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14\") " pod="openshift-monitoring/node-exporter-vh7rl"
Nov 01 05:03:31.998916 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:31.998898    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 01 05:03:32.000336 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:32.000320    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-5hwhj"]
Nov 01 05:03:32.008567 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:32.008550    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Nov 01 05:03:32.052678 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:32.052608    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-btxwf\""
Nov 01 05:03:32.052678 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:32.052614    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Nov 01 05:03:32.052678 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:32.052652    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Nov 01 05:03:32.098139 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:32.098114    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/805c87a3-63e9-4c13-9c90-c05f73282edc-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-5hwhj\" (UID: \"805c87a3-63e9-4c13-9c90-c05f73282edc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5hwhj"
Nov 01 05:03:32.098247 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:32.098147    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14-node-exporter-textfile\") pod \"node-exporter-vh7rl\" (UID: \"9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14\") " pod="openshift-monitoring/node-exporter-vh7rl"
Nov 01 05:03:32.098247 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:32.098165    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14-node-exporter-wtmp\") pod \"node-exporter-vh7rl\" (UID: \"9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14\") " pod="openshift-monitoring/node-exporter-vh7rl"
Nov 01 05:03:32.098247 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:32.098184    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14-sys\") pod \"node-exporter-vh7rl\" (UID: \"9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14\") " pod="openshift-monitoring/node-exporter-vh7rl"
Nov 01 05:03:32.098247 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:32.098240    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14-sys\") pod \"node-exporter-vh7rl\" (UID: \"9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14\") " pod="openshift-monitoring/node-exporter-vh7rl"
Nov 01 05:03:32.098479 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:32.098274    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14-root\") pod \"node-exporter-vh7rl\" (UID: \"9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14\") " pod="openshift-monitoring/node-exporter-vh7rl"
Nov 01 05:03:32.098479 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:32.098296    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14-node-exporter-tls\") pod \"node-exporter-vh7rl\" (UID: \"9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14\") " pod="openshift-monitoring/node-exporter-vh7rl"
Nov 01 05:03:32.098479 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:32.098324    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/805c87a3-63e9-4c13-9c90-c05f73282edc-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-5hwhj\" (UID: \"805c87a3-63e9-4c13-9c90-c05f73282edc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5hwhj"
Nov 01 05:03:32.098479 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:32.098353    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vh7rl\" (UID: \"9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14\") " pod="openshift-monitoring/node-exporter-vh7rl"
Nov 01 05:03:32.098479 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:32.098357    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14-root\") pod \"node-exporter-vh7rl\" (UID: \"9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14\") " pod="openshift-monitoring/node-exporter-vh7rl"
Nov 01 05:03:32.098479 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:32.098381    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvgxl\" (UniqueName: \"kubernetes.io/projected/805c87a3-63e9-4c13-9c90-c05f73282edc-kube-api-access-hvgxl\") pod \"kube-state-metrics-5dc8d7d89b-5hwhj\" (UID: \"805c87a3-63e9-4c13-9c90-c05f73282edc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5hwhj"
Nov 01 05:03:32.098479 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:32.098320    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14-node-exporter-wtmp\") pod \"node-exporter-vh7rl\" (UID: \"9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14\") " pod="openshift-monitoring/node-exporter-vh7rl"
Nov 01 05:03:32.098800 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:32.098484    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14-metrics-client-ca\") pod \"node-exporter-vh7rl\" (UID: \"9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14\") " pod="openshift-monitoring/node-exporter-vh7rl"
Nov 01 05:03:32.098800 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:32.098526    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/805c87a3-63e9-4c13-9c90-c05f73282edc-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-5hwhj\" (UID: \"805c87a3-63e9-4c13-9c90-c05f73282edc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5hwhj"
Nov 01 05:03:32.098800 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:32.098553    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14-node-exporter-textfile\") pod \"node-exporter-vh7rl\" (UID: \"9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14\") " pod="openshift-monitoring/node-exporter-vh7rl"
Nov 01 05:03:32.098800 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:32.098556    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/805c87a3-63e9-4c13-9c90-c05f73282edc-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-5hwhj\" (UID: \"805c87a3-63e9-4c13-9c90-c05f73282edc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5hwhj"
Nov 01 05:03:32.098800 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:32.098573    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/805c87a3-63e9-4c13-9c90-c05f73282edc-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-5hwhj\" (UID: \"805c87a3-63e9-4c13-9c90-c05f73282edc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5hwhj"
Nov 01 05:03:32.098800 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:32.098601    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/805c87a3-63e9-4c13-9c90-c05f73282edc-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-5hwhj\" (UID: \"805c87a3-63e9-4c13-9c90-c05f73282edc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5hwhj"
Nov 01 05:03:32.098800 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:32.098653    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5zxr\" (UniqueName: \"kubernetes.io/projected/9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14-kube-api-access-r5zxr\") pod \"node-exporter-vh7rl\" (UID: \"9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14\") " pod="openshift-monitoring/node-exporter-vh7rl"
Nov 01 05:03:32.098800 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:32.098692    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14-node-exporter-accelerators-collector-config\") pod \"node-exporter-vh7rl\" (UID: \"9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14\") " pod="openshift-monitoring/node-exporter-vh7rl"
Nov 01 05:03:32.099198 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:32.099099    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14-metrics-client-ca\") pod \"node-exporter-vh7rl\" (UID: \"9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14\") " pod="openshift-monitoring/node-exporter-vh7rl"
Nov 01 05:03:32.099254 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:32.099222    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14-node-exporter-accelerators-collector-config\") pod \"node-exporter-vh7rl\" (UID: \"9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14\") " pod="openshift-monitoring/node-exporter-vh7rl"
Nov 01 05:03:32.099344 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:32.099323    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/805c87a3-63e9-4c13-9c90-c05f73282edc-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-5hwhj\" (UID: \"805c87a3-63e9-4c13-9c90-c05f73282edc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5hwhj"
Nov 01 05:03:32.099886 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:32.099857    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/805c87a3-63e9-4c13-9c90-c05f73282edc-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-5hwhj\" (UID: \"805c87a3-63e9-4c13-9c90-c05f73282edc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5hwhj"
Nov 01 05:03:32.101232 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:32.101202    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vh7rl\" (UID: \"9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14\") " pod="openshift-monitoring/node-exporter-vh7rl"
Nov 01 05:03:32.101582 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:32.101555    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14-node-exporter-tls\") pod \"node-exporter-vh7rl\" (UID: \"9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14\") " pod="openshift-monitoring/node-exporter-vh7rl"
Nov 01 05:03:32.101754 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:32.101734    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/805c87a3-63e9-4c13-9c90-c05f73282edc-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-5hwhj\" (UID: \"805c87a3-63e9-4c13-9c90-c05f73282edc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5hwhj"
Nov 01 05:03:32.103388 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:32.103370    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/805c87a3-63e9-4c13-9c90-c05f73282edc-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-5hwhj\" (UID: \"805c87a3-63e9-4c13-9c90-c05f73282edc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5hwhj"
Nov 01 05:03:32.166323 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:32.165578    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5zxr\" (UniqueName: \"kubernetes.io/projected/9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14-kube-api-access-r5zxr\") pod \"node-exporter-vh7rl\" (UID: \"9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14\") " pod="openshift-monitoring/node-exporter-vh7rl"
Nov 01 05:03:32.166750 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:32.166515    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvgxl\" (UniqueName: \"kubernetes.io/projected/805c87a3-63e9-4c13-9c90-c05f73282edc-kube-api-access-hvgxl\") pod \"kube-state-metrics-5dc8d7d89b-5hwhj\" (UID: \"805c87a3-63e9-4c13-9c90-c05f73282edc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5hwhj"
Nov 01 05:03:32.192917 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:32.192892    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-645d98cc97-fdph2"
Nov 01 05:03:32.193062 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:32.192938    2571 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-645d98cc97-fdph2"
Nov 01 05:03:32.198119 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:32.198085    2571 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-645d98cc97-fdph2"
Nov 01 05:03:32.212634 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:32.212614    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vh7rl"
Nov 01 05:03:32.219770 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:32.219621    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5hwhj"
Nov 01 05:03:32.226369 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:03:32.226345    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c9e2fb8_c0e2_4cfa_ac00_811559e4ef14.slice/crio-6f360fd45dad65dd6e7d7d898de884c382f2d686f42036ee1b5d1f749bb636d9 WatchSource:0}: Error finding container 6f360fd45dad65dd6e7d7d898de884c382f2d686f42036ee1b5d1f749bb636d9: Status 404 returned error can't find the container with id 6f360fd45dad65dd6e7d7d898de884c382f2d686f42036ee1b5d1f749bb636d9
Nov 01 05:03:32.256847 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:32.256821    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:32.513917 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:32.513884    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:32.644496 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:32.644438    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vh7rl" event={"ID":"9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14","Type":"ContainerStarted","Data":"6f360fd45dad65dd6e7d7d898de884c382f2d686f42036ee1b5d1f749bb636d9"}
Nov 01 05:03:32.649000 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:32.648973    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-645d98cc97-fdph2"
Nov 01 05:03:32.655060 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:32.655035    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-5hwhj"]
Nov 01 05:03:32.658737 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:03:32.658709    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod805c87a3_63e9_4c13_9c90_c05f73282edc.slice/crio-a6cb025b09c74b2375547c39e2bcf11018d0fe067f47fa5bc687270d83d3c4d5 WatchSource:0}: Error finding container a6cb025b09c74b2375547c39e2bcf11018d0fe067f47fa5bc687270d83d3c4d5: Status 404 returned error can't find the container with id a6cb025b09c74b2375547c39e2bcf11018d0fe067f47fa5bc687270d83d3c4d5
Nov 01 05:03:32.747168 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:32.747134    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-76cf6f4bb6-dwvbm"]
Nov 01 05:03:32.751656 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:32.751631    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76cf6f4bb6-dwvbm"
Nov 01 05:03:32.804996 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:32.804916    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1bd1166d-68e9-4035-b920-e42f5bf05d11-console-serving-cert\") pod \"console-76cf6f4bb6-dwvbm\" (UID: \"1bd1166d-68e9-4035-b920-e42f5bf05d11\") " pod="openshift-console/console-76cf6f4bb6-dwvbm"
Nov 01 05:03:32.804996 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:32.804961    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1bd1166d-68e9-4035-b920-e42f5bf05d11-tmp\") pod \"console-76cf6f4bb6-dwvbm\" (UID: \"1bd1166d-68e9-4035-b920-e42f5bf05d11\") " pod="openshift-console/console-76cf6f4bb6-dwvbm"
Nov 01 05:03:32.805166 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:32.805074    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1bd1166d-68e9-4035-b920-e42f5bf05d11-service-ca\") pod \"console-76cf6f4bb6-dwvbm\" (UID: \"1bd1166d-68e9-4035-b920-e42f5bf05d11\") " pod="openshift-console/console-76cf6f4bb6-dwvbm"
Nov 01 05:03:32.805166 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:32.805111    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1bd1166d-68e9-4035-b920-e42f5bf05d11-console-config\") pod \"console-76cf6f4bb6-dwvbm\" (UID: \"1bd1166d-68e9-4035-b920-e42f5bf05d11\") " pod="openshift-console/console-76cf6f4bb6-dwvbm"
Nov 01 05:03:32.805166 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:32.805149    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1bd1166d-68e9-4035-b920-e42f5bf05d11-console-oauth-config\") pod \"console-76cf6f4bb6-dwvbm\" (UID: \"1bd1166d-68e9-4035-b920-e42f5bf05d11\") " pod="openshift-console/console-76cf6f4bb6-dwvbm"
Nov 01 05:03:32.805276 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:32.805213    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1bd1166d-68e9-4035-b920-e42f5bf05d11-oauth-serving-cert\") pod \"console-76cf6f4bb6-dwvbm\" (UID: \"1bd1166d-68e9-4035-b920-e42f5bf05d11\") " pod="openshift-console/console-76cf6f4bb6-dwvbm"
Nov 01 05:03:32.805324 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:32.805271    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1bd1166d-68e9-4035-b920-e42f5bf05d11-trusted-ca-bundle\") pod \"console-76cf6f4bb6-dwvbm\" (UID: \"1bd1166d-68e9-4035-b920-e42f5bf05d11\") " pod="openshift-console/console-76cf6f4bb6-dwvbm"
Nov 01 05:03:32.805431 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:32.805409    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7npk7\" (UniqueName: \"kubernetes.io/projected/1bd1166d-68e9-4035-b920-e42f5bf05d11-kube-api-access-7npk7\") pod \"console-76cf6f4bb6-dwvbm\" (UID: \"1bd1166d-68e9-4035-b920-e42f5bf05d11\") " pod="openshift-console/console-76cf6f4bb6-dwvbm"
Nov 01 05:03:32.856406 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:32.856365    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Nov 01 05:03:32.858198 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:32.858172    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76cf6f4bb6-dwvbm"]
Nov 01 05:03:32.906683 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:32.906647    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7npk7\" (UniqueName: \"kubernetes.io/projected/1bd1166d-68e9-4035-b920-e42f5bf05d11-kube-api-access-7npk7\") pod \"console-76cf6f4bb6-dwvbm\" (UID: \"1bd1166d-68e9-4035-b920-e42f5bf05d11\") " pod="openshift-console/console-76cf6f4bb6-dwvbm"
Nov 01 05:03:32.906831 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:32.906703    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1bd1166d-68e9-4035-b920-e42f5bf05d11-console-serving-cert\") pod \"console-76cf6f4bb6-dwvbm\" (UID: \"1bd1166d-68e9-4035-b920-e42f5bf05d11\") " pod="openshift-console/console-76cf6f4bb6-dwvbm"
Nov 01 05:03:32.906831 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:32.906735    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1bd1166d-68e9-4035-b920-e42f5bf05d11-tmp\") pod \"console-76cf6f4bb6-dwvbm\" (UID: \"1bd1166d-68e9-4035-b920-e42f5bf05d11\") " pod="openshift-console/console-76cf6f4bb6-dwvbm"
Nov 01 05:03:32.906831 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:32.906782    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1bd1166d-68e9-4035-b920-e42f5bf05d11-service-ca\") pod \"console-76cf6f4bb6-dwvbm\" (UID: \"1bd1166d-68e9-4035-b920-e42f5bf05d11\") " pod="openshift-console/console-76cf6f4bb6-dwvbm"
Nov 01 05:03:32.906831 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:32.906809    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1bd1166d-68e9-4035-b920-e42f5bf05d11-console-config\") pod \"console-76cf6f4bb6-dwvbm\" (UID: \"1bd1166d-68e9-4035-b920-e42f5bf05d11\") " pod="openshift-console/console-76cf6f4bb6-dwvbm"
Nov 01 05:03:32.907049 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:32.906839    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1bd1166d-68e9-4035-b920-e42f5bf05d11-console-oauth-config\") pod \"console-76cf6f4bb6-dwvbm\" (UID: \"1bd1166d-68e9-4035-b920-e42f5bf05d11\") " pod="openshift-console/console-76cf6f4bb6-dwvbm"
Nov 01 05:03:32.907049 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:32.906885    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1bd1166d-68e9-4035-b920-e42f5bf05d11-oauth-serving-cert\") pod \"console-76cf6f4bb6-dwvbm\" (UID: \"1bd1166d-68e9-4035-b920-e42f5bf05d11\") " pod="openshift-console/console-76cf6f4bb6-dwvbm"
Nov 01 05:03:32.907049 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:32.906914    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1bd1166d-68e9-4035-b920-e42f5bf05d11-trusted-ca-bundle\") pod \"console-76cf6f4bb6-dwvbm\" (UID: \"1bd1166d-68e9-4035-b920-e42f5bf05d11\") " pod="openshift-console/console-76cf6f4bb6-dwvbm"
Nov 01 05:03:32.907193 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:32.907147    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1bd1166d-68e9-4035-b920-e42f5bf05d11-tmp\") pod \"console-76cf6f4bb6-dwvbm\" (UID: \"1bd1166d-68e9-4035-b920-e42f5bf05d11\") " pod="openshift-console/console-76cf6f4bb6-dwvbm"
Nov 01 05:03:32.907533 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:32.907506    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1bd1166d-68e9-4035-b920-e42f5bf05d11-service-ca\") pod \"console-76cf6f4bb6-dwvbm\" (UID: \"1bd1166d-68e9-4035-b920-e42f5bf05d11\") " pod="openshift-console/console-76cf6f4bb6-dwvbm"
Nov 01 05:03:32.907864 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:32.907842    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1bd1166d-68e9-4035-b920-e42f5bf05d11-trusted-ca-bundle\") pod \"console-76cf6f4bb6-dwvbm\" (UID: \"1bd1166d-68e9-4035-b920-e42f5bf05d11\") " pod="openshift-console/console-76cf6f4bb6-dwvbm"
Nov 01 05:03:32.907941 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:32.907925    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1bd1166d-68e9-4035-b920-e42f5bf05d11-console-config\") pod \"console-76cf6f4bb6-dwvbm\" (UID: \"1bd1166d-68e9-4035-b920-e42f5bf05d11\") " pod="openshift-console/console-76cf6f4bb6-dwvbm"
Nov 01 05:03:32.908032 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:32.908011    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1bd1166d-68e9-4035-b920-e42f5bf05d11-oauth-serving-cert\") pod \"console-76cf6f4bb6-dwvbm\" (UID: \"1bd1166d-68e9-4035-b920-e42f5bf05d11\") " pod="openshift-console/console-76cf6f4bb6-dwvbm"
Nov 01 05:03:32.909551 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:32.909533    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1bd1166d-68e9-4035-b920-e42f5bf05d11-console-serving-cert\") pod \"console-76cf6f4bb6-dwvbm\" (UID: \"1bd1166d-68e9-4035-b920-e42f5bf05d11\") " pod="openshift-console/console-76cf6f4bb6-dwvbm"
Nov 01 05:03:32.909714 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:32.909689    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1bd1166d-68e9-4035-b920-e42f5bf05d11-console-oauth-config\") pod \"console-76cf6f4bb6-dwvbm\" (UID: \"1bd1166d-68e9-4035-b920-e42f5bf05d11\") " pod="openshift-console/console-76cf6f4bb6-dwvbm"
Nov 01 05:03:32.969084 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:32.969053    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7npk7\" (UniqueName: \"kubernetes.io/projected/1bd1166d-68e9-4035-b920-e42f5bf05d11-kube-api-access-7npk7\") pod \"console-76cf6f4bb6-dwvbm\" (UID: \"1bd1166d-68e9-4035-b920-e42f5bf05d11\") " pod="openshift-console/console-76cf6f4bb6-dwvbm"
Nov 01 05:03:33.065592 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:33.065521    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76cf6f4bb6-dwvbm"
Nov 01 05:03:33.230370 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:33.230250    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:33.313027 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:33.312993    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4321e0ae-e339-40f2-875a-941b1183e766-metrics-certs\") pod \"network-metrics-daemon-267rv\" (UID: \"4321e0ae-e339-40f2-875a-941b1183e766\") " pod="openshift-multus/network-metrics-daemon-267rv"
Nov 01 05:03:33.313200 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:33.313081    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e856df5a-5c5e-4421-ae7d-80a760bc72e2-original-pull-secret\") pod \"global-pull-secret-syncer-9pcxf\" (UID: \"e856df5a-5c5e-4421-ae7d-80a760bc72e2\") " pod="kube-system/global-pull-secret-syncer-9pcxf"
Nov 01 05:03:33.365776 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:33.365691    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 01 05:03:33.365776 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:33.365715    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 01 05:03:33.386859 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:33.386826    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e856df5a-5c5e-4421-ae7d-80a760bc72e2-original-pull-secret\") pod \"global-pull-secret-syncer-9pcxf\" (UID: \"e856df5a-5c5e-4421-ae7d-80a760bc72e2\") " pod="kube-system/global-pull-secret-syncer-9pcxf"
Nov 01 05:03:33.388831 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:33.388796    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4321e0ae-e339-40f2-875a-941b1183e766-metrics-certs\") pod \"network-metrics-daemon-267rv\" (UID: \"4321e0ae-e339-40f2-875a-941b1183e766\") " pod="openshift-multus/network-metrics-daemon-267rv"
Nov 01 05:03:33.450486 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:33.450437    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76cf6f4bb6-dwvbm"]
Nov 01 05:03:33.455307 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:03:33.455271    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bd1166d_68e9_4035_b920_e42f5bf05d11.slice/crio-94b090bb2f81614395a6c92d20a127033288b6317a3746960c12ca9c06f937c8 WatchSource:0}: Error finding container 94b090bb2f81614395a6c92d20a127033288b6317a3746960c12ca9c06f937c8: Status 404 returned error can't find the container with id 94b090bb2f81614395a6c92d20a127033288b6317a3746960c12ca9c06f937c8
Nov 01 05:03:33.498240 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:33.498211    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:33.508606 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:33.508585    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6k7br\""
Nov 01 05:03:33.517542 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:33.517521    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-267rv"
Nov 01 05:03:33.549680 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:33.549656    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-g7ktw\""
Nov 01 05:03:33.558500 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:33.558423    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9pcxf"
Nov 01 05:03:33.651871 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:33.651747    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5hwhj" event={"ID":"805c87a3-63e9-4c13-9c90-c05f73282edc","Type":"ContainerStarted","Data":"a6cb025b09c74b2375547c39e2bcf11018d0fe067f47fa5bc687270d83d3c4d5"}
Nov 01 05:03:33.657704 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:33.657541    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76cf6f4bb6-dwvbm" event={"ID":"1bd1166d-68e9-4035-b920-e42f5bf05d11","Type":"ContainerStarted","Data":"c7ab7d992cbad4554a3f823f7d45f7129516fc079ac30429bb2b139a4c635ffc"}
Nov 01 05:03:33.657704 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:33.657586    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76cf6f4bb6-dwvbm" event={"ID":"1bd1166d-68e9-4035-b920-e42f5bf05d11","Type":"ContainerStarted","Data":"94b090bb2f81614395a6c92d20a127033288b6317a3746960c12ca9c06f937c8"}
Nov 01 05:03:33.940489 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:33.940418    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-76cf6f4bb6-dwvbm" podStartSLOduration=1.940402047 podStartE2EDuration="1.940402047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:03:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:03:33.777730295 +0000 UTC m=+65.978627646" watchObservedRunningTime="2026-11-01 05:03:33.940402047 +0000 UTC m=+66.141299392"
Nov 01 05:03:33.940971 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:33.940898    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-267rv"]
Nov 01 05:03:33.944114 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:03:33.944085    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4321e0ae_e339_40f2_875a_941b1183e766.slice/crio-e4b4d6c6a74ba41ccfa800406dd6347bda703d9e8cde5971cf0cf9f4b1d1e118 WatchSource:0}: Error finding container e4b4d6c6a74ba41ccfa800406dd6347bda703d9e8cde5971cf0cf9f4b1d1e118: Status 404 returned error can't find the container with id e4b4d6c6a74ba41ccfa800406dd6347bda703d9e8cde5971cf0cf9f4b1d1e118
Nov 01 05:03:33.989370 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:33.989333    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9pcxf"]
Nov 01 05:03:33.993573 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:03:33.993543    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode856df5a_5c5e_4421_ae7d_80a760bc72e2.slice/crio-941920de309566d6056dbe93be54c9881a2d6a53c9d51e866cf373cb386906f9 WatchSource:0}: Error finding container 941920de309566d6056dbe93be54c9881a2d6a53c9d51e866cf373cb386906f9: Status 404 returned error can't find the container with id 941920de309566d6056dbe93be54c9881a2d6a53c9d51e866cf373cb386906f9
Nov 01 05:03:34.128014 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:34.127958    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:34.418351 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:34.418316    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:34.535222 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:34.535140    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-77ddfd4d86-btjkv"]
Nov 01 05:03:34.556786 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:34.556757    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-77ddfd4d86-btjkv"
Nov 01 05:03:34.613739 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:34.613698    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-77ddfd4d86-btjkv"]
Nov 01 05:03:34.626082 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:34.626048    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/80c10aac-5175-46d6-a18b-109169a2461b-secret-thanos-querier-tls\") pod \"thanos-querier-77ddfd4d86-btjkv\" (UID: \"80c10aac-5175-46d6-a18b-109169a2461b\") " pod="openshift-monitoring/thanos-querier-77ddfd4d86-btjkv"
Nov 01 05:03:34.626211 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:34.626092    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/80c10aac-5175-46d6-a18b-109169a2461b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-77ddfd4d86-btjkv\" (UID: \"80c10aac-5175-46d6-a18b-109169a2461b\") " pod="openshift-monitoring/thanos-querier-77ddfd4d86-btjkv"
Nov 01 05:03:34.626211 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:34.626117    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/80c10aac-5175-46d6-a18b-109169a2461b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-77ddfd4d86-btjkv\" (UID: \"80c10aac-5175-46d6-a18b-109169a2461b\") " pod="openshift-monitoring/thanos-querier-77ddfd4d86-btjkv"
Nov 01 05:03:34.626211 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:34.626133    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/80c10aac-5175-46d6-a18b-109169a2461b-secret-grpc-tls\") pod \"thanos-querier-77ddfd4d86-btjkv\" (UID: \"80c10aac-5175-46d6-a18b-109169a2461b\") " pod="openshift-monitoring/thanos-querier-77ddfd4d86-btjkv"
Nov 01 05:03:34.626211 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:34.626159    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/80c10aac-5175-46d6-a18b-109169a2461b-metrics-client-ca\") pod \"thanos-querier-77ddfd4d86-btjkv\" (UID: \"80c10aac-5175-46d6-a18b-109169a2461b\") " pod="openshift-monitoring/thanos-querier-77ddfd4d86-btjkv"
Nov 01 05:03:34.626211 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:34.626175    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/80c10aac-5175-46d6-a18b-109169a2461b-tmp\") pod \"thanos-querier-77ddfd4d86-btjkv\" (UID: \"80c10aac-5175-46d6-a18b-109169a2461b\") " pod="openshift-monitoring/thanos-querier-77ddfd4d86-btjkv"
Nov 01 05:03:34.626211 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:34.626195    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/80c10aac-5175-46d6-a18b-109169a2461b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-77ddfd4d86-btjkv\" (UID: \"80c10aac-5175-46d6-a18b-109169a2461b\") " pod="openshift-monitoring/thanos-querier-77ddfd4d86-btjkv"
Nov 01 05:03:34.626211 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:34.626223    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7hhr\" (UniqueName: \"kubernetes.io/projected/80c10aac-5175-46d6-a18b-109169a2461b-kube-api-access-c7hhr\") pod \"thanos-querier-77ddfd4d86-btjkv\" (UID: \"80c10aac-5175-46d6-a18b-109169a2461b\") " pod="openshift-monitoring/thanos-querier-77ddfd4d86-btjkv"
Nov 01 05:03:34.626483 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:34.626249    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/80c10aac-5175-46d6-a18b-109169a2461b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-77ddfd4d86-btjkv\" (UID: \"80c10aac-5175-46d6-a18b-109169a2461b\") " pod="openshift-monitoring/thanos-querier-77ddfd4d86-btjkv"
Nov 01 05:03:34.647680 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:34.647647    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:34.649293 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:34.649261    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Nov 01 05:03:34.649412 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:34.649332    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Nov 01 05:03:34.649537 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:34.649433    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Nov 01 05:03:34.649537 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:34.649478    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Nov 01 05:03:34.649537 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:34.649493    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Nov 01 05:03:34.649711 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:34.649550    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-jtxhq\""
Nov 01 05:03:34.649711 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:34.649616    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-b2kibd754ifqu\""
Nov 01 05:03:34.662241 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:34.662137    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-267rv" event={"ID":"4321e0ae-e339-40f2-875a-941b1183e766","Type":"ContainerStarted","Data":"e4b4d6c6a74ba41ccfa800406dd6347bda703d9e8cde5971cf0cf9f4b1d1e118"}
Nov 01 05:03:34.664238 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:34.664215    2571 generic.go:356] "Generic (PLEG): container finished" podID="9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14" containerID="8e68c281df88a0c78cd55be82559c64946ed15163a4298d5f7fdae9720f71258" exitCode=0
Nov 01 05:03:34.664557 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:34.664529    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vh7rl" event={"ID":"9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14","Type":"ContainerDied","Data":"8e68c281df88a0c78cd55be82559c64946ed15163a4298d5f7fdae9720f71258"}
Nov 01 05:03:34.665900 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:34.665859    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9pcxf" event={"ID":"e856df5a-5c5e-4421-ae7d-80a760bc72e2","Type":"ContainerStarted","Data":"941920de309566d6056dbe93be54c9881a2d6a53c9d51e866cf373cb386906f9"}
Nov 01 05:03:34.727179 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:34.727155    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/80c10aac-5175-46d6-a18b-109169a2461b-metrics-client-ca\") pod \"thanos-querier-77ddfd4d86-btjkv\" (UID: \"80c10aac-5175-46d6-a18b-109169a2461b\") " pod="openshift-monitoring/thanos-querier-77ddfd4d86-btjkv"
Nov 01 05:03:34.727316 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:34.727188    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/80c10aac-5175-46d6-a18b-109169a2461b-tmp\") pod \"thanos-querier-77ddfd4d86-btjkv\" (UID: \"80c10aac-5175-46d6-a18b-109169a2461b\") " pod="openshift-monitoring/thanos-querier-77ddfd4d86-btjkv"
Nov 01 05:03:34.727316 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:34.727249    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/80c10aac-5175-46d6-a18b-109169a2461b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-77ddfd4d86-btjkv\" (UID: \"80c10aac-5175-46d6-a18b-109169a2461b\") " pod="openshift-monitoring/thanos-querier-77ddfd4d86-btjkv"
Nov 01 05:03:34.727424 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:34.727336    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7hhr\" (UniqueName: \"kubernetes.io/projected/80c10aac-5175-46d6-a18b-109169a2461b-kube-api-access-c7hhr\") pod \"thanos-querier-77ddfd4d86-btjkv\" (UID: \"80c10aac-5175-46d6-a18b-109169a2461b\") " pod="openshift-monitoring/thanos-querier-77ddfd4d86-btjkv"
Nov 01 05:03:34.727424 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:34.727404    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/80c10aac-5175-46d6-a18b-109169a2461b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-77ddfd4d86-btjkv\" (UID: \"80c10aac-5175-46d6-a18b-109169a2461b\") " pod="openshift-monitoring/thanos-querier-77ddfd4d86-btjkv"
Nov 01 05:03:34.728201 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:34.727434    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/80c10aac-5175-46d6-a18b-109169a2461b-secret-thanos-querier-tls\") pod \"thanos-querier-77ddfd4d86-btjkv\" (UID: \"80c10aac-5175-46d6-a18b-109169a2461b\") " pod="openshift-monitoring/thanos-querier-77ddfd4d86-btjkv"
Nov 01 05:03:34.728201 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:34.727490    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/80c10aac-5175-46d6-a18b-109169a2461b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-77ddfd4d86-btjkv\" (UID: \"80c10aac-5175-46d6-a18b-109169a2461b\") " pod="openshift-monitoring/thanos-querier-77ddfd4d86-btjkv"
Nov 01 05:03:34.728201 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:34.727519    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/80c10aac-5175-46d6-a18b-109169a2461b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-77ddfd4d86-btjkv\" (UID: \"80c10aac-5175-46d6-a18b-109169a2461b\") " pod="openshift-monitoring/thanos-querier-77ddfd4d86-btjkv"
Nov 01 05:03:34.728201 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:34.727607    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/80c10aac-5175-46d6-a18b-109169a2461b-secret-grpc-tls\") pod \"thanos-querier-77ddfd4d86-btjkv\" (UID: \"80c10aac-5175-46d6-a18b-109169a2461b\") " pod="openshift-monitoring/thanos-querier-77ddfd4d86-btjkv"
Nov 01 05:03:34.728201 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:34.727653    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/80c10aac-5175-46d6-a18b-109169a2461b-tmp\") pod \"thanos-querier-77ddfd4d86-btjkv\" (UID: \"80c10aac-5175-46d6-a18b-109169a2461b\") " pod="openshift-monitoring/thanos-querier-77ddfd4d86-btjkv"
Nov 01 05:03:34.728201 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:34.727972    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/80c10aac-5175-46d6-a18b-109169a2461b-metrics-client-ca\") pod \"thanos-querier-77ddfd4d86-btjkv\" (UID: \"80c10aac-5175-46d6-a18b-109169a2461b\") " pod="openshift-monitoring/thanos-querier-77ddfd4d86-btjkv"
Nov 01 05:03:34.730279 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:34.730252    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/80c10aac-5175-46d6-a18b-109169a2461b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-77ddfd4d86-btjkv\" (UID: \"80c10aac-5175-46d6-a18b-109169a2461b\") " pod="openshift-monitoring/thanos-querier-77ddfd4d86-btjkv"
Nov 01 05:03:34.731014 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:34.730987    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/80c10aac-5175-46d6-a18b-109169a2461b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-77ddfd4d86-btjkv\" (UID: \"80c10aac-5175-46d6-a18b-109169a2461b\") " pod="openshift-monitoring/thanos-querier-77ddfd4d86-btjkv"
Nov 01 05:03:34.731014 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:34.731007    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/80c10aac-5175-46d6-a18b-109169a2461b-secret-grpc-tls\") pod \"thanos-querier-77ddfd4d86-btjkv\" (UID: \"80c10aac-5175-46d6-a18b-109169a2461b\") " pod="openshift-monitoring/thanos-querier-77ddfd4d86-btjkv"
Nov 01 05:03:34.731237 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:34.731214    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/80c10aac-5175-46d6-a18b-109169a2461b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-77ddfd4d86-btjkv\" (UID: \"80c10aac-5175-46d6-a18b-109169a2461b\") " pod="openshift-monitoring/thanos-querier-77ddfd4d86-btjkv"
Nov 01 05:03:34.731272 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:34.731239    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/80c10aac-5175-46d6-a18b-109169a2461b-secret-thanos-querier-tls\") pod \"thanos-querier-77ddfd4d86-btjkv\" (UID: \"80c10aac-5175-46d6-a18b-109169a2461b\") " pod="openshift-monitoring/thanos-querier-77ddfd4d86-btjkv"
Nov 01 05:03:34.731313 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:34.731272    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/80c10aac-5175-46d6-a18b-109169a2461b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-77ddfd4d86-btjkv\" (UID: \"80c10aac-5175-46d6-a18b-109169a2461b\") " pod="openshift-monitoring/thanos-querier-77ddfd4d86-btjkv"
Nov 01 05:03:34.790886 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:34.790835    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7hhr\" (UniqueName: \"kubernetes.io/projected/80c10aac-5175-46d6-a18b-109169a2461b-kube-api-access-c7hhr\") pod \"thanos-querier-77ddfd4d86-btjkv\" (UID: \"80c10aac-5175-46d6-a18b-109169a2461b\") " pod="openshift-monitoring/thanos-querier-77ddfd4d86-btjkv"
Nov 01 05:03:34.838550 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:34.838518    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:34.949651 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:34.949618    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-77ddfd4d86-btjkv"
Nov 01 05:03:35.571595 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:35.571563    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-77ddfd4d86-btjkv"]
Nov 01 05:03:35.624691 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:03:35.624657    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80c10aac_5175_46d6_a18b_109169a2461b.slice/crio-a4e2086e72083ac7db69643e5261a496346ab4a08e69339adc535572a63c94f4 WatchSource:0}: Error finding container a4e2086e72083ac7db69643e5261a496346ab4a08e69339adc535572a63c94f4: Status 404 returned error can't find the container with id a4e2086e72083ac7db69643e5261a496346ab4a08e69339adc535572a63c94f4
Nov 01 05:03:35.674071 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:35.674041    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77ddfd4d86-btjkv" event={"ID":"80c10aac-5175-46d6-a18b-109169a2461b","Type":"ContainerStarted","Data":"a4e2086e72083ac7db69643e5261a496346ab4a08e69339adc535572a63c94f4"}
Nov 01 05:03:35.677737 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:35.677706    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vh7rl" event={"ID":"9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14","Type":"ContainerStarted","Data":"12a8cb257ed7f234dd9e50b9b1315786769469f49635b67f8d21878088c6b00e"}
Nov 01 05:03:35.677871 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:35.677762    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vh7rl" event={"ID":"9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14","Type":"ContainerStarted","Data":"ea67502bdfdaccccf3f6c92ce966db12df00589bd67f08c94cdc5240ee041824"}
Nov 01 05:03:35.682439 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:35.682369    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5hwhj" event={"ID":"805c87a3-63e9-4c13-9c90-c05f73282edc","Type":"ContainerStarted","Data":"0a36dfe6d03fd90886c856880f74217cf4d458bdf95d03fcfa7959e223a4fae8"}
Nov 01 05:03:35.682439 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:35.682416    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5hwhj" event={"ID":"805c87a3-63e9-4c13-9c90-c05f73282edc","Type":"ContainerStarted","Data":"46c59d93f8e377ec4a850425c84a834a0ab8f46146d0b567e741e80e395d6d05"}
Nov 01 05:03:35.682580 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:35.682481    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5hwhj" event={"ID":"805c87a3-63e9-4c13-9c90-c05f73282edc","Type":"ContainerStarted","Data":"b1e2d22ea005a9987106e5dfa3253b4d5eabb31c0bb19ef6e8cb8582829337a1"}
Nov 01 05:03:35.801513 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:35.800961    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-vh7rl" podStartSLOduration=2.662566079 podStartE2EDuration="4.800943955s" totalImagesPullingTime="2.033011454s" totalInitContainerRuntime="105.366422ms" isStatefulPod=true podCreationTimestamp="2026-11-01 05:03:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:03:35.799430413 +0000 UTC m=+68.000327762" watchObservedRunningTime="2026-11-01 05:03:35.800943955 +0000 UTC m=+68.001841331"
Nov 01 05:03:35.925078 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:35.925013    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5hwhj" podStartSLOduration=2.504351904 podStartE2EDuration="4.924995299s" totalImagesPullingTime="2.420643395s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:03:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:03:35.924658362 +0000 UTC m=+68.125555716" watchObservedRunningTime="2026-11-01 05:03:35.924995299 +0000 UTC m=+68.125892651"
Nov 01 05:03:36.047849 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:36.047796    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:36.324714 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:36.324646    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:36.505760 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:36.505715    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:36.692117 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:36.692064    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-267rv" event={"ID":"4321e0ae-e339-40f2-875a-941b1183e766","Type":"ContainerStarted","Data":"ecc93e69bcc90c0b4e7c63f1fc900c1b6d05324d16671a424059fb71031af3e3"}
Nov 01 05:03:36.749365 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:36.749329    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:36.822703 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:36.822670    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7b6db58fc-994xl"]
Nov 01 05:03:36.845279 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:36.845241    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7b6db58fc-994xl"
Nov 01 05:03:36.887315 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:36.887186    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7b6db58fc-994xl"]
Nov 01 05:03:36.903188 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:36.903159    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 01 05:03:36.906254 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:36.906229    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Nov 01 05:03:36.922048 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:36.922026    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Nov 01 05:03:36.922330 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:36.922242    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-zqvfm\""
Nov 01 05:03:36.922330 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:36.922344    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Nov 01 05:03:36.922558 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:36.922419    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-b8lb60f3fpm0o\""
Nov 01 05:03:36.948861 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:36.948776    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:36.950782 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:36.950756    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6-secret-metrics-server-tls\") pod \"metrics-server-7b6db58fc-994xl\" (UID: \"0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6\") " pod="openshift-monitoring/metrics-server-7b6db58fc-994xl"
Nov 01 05:03:36.950897 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:36.950807    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6-client-ca-bundle\") pod \"metrics-server-7b6db58fc-994xl\" (UID: \"0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6\") " pod="openshift-monitoring/metrics-server-7b6db58fc-994xl"
Nov 01 05:03:36.950897 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:36.950858    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6-secret-metrics-server-client-certs\") pod \"metrics-server-7b6db58fc-994xl\" (UID: \"0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6\") " pod="openshift-monitoring/metrics-server-7b6db58fc-994xl"
Nov 01 05:03:36.951003 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:36.950896    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6-audit-log\") pod \"metrics-server-7b6db58fc-994xl\" (UID: \"0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6\") " pod="openshift-monitoring/metrics-server-7b6db58fc-994xl"
Nov 01 05:03:36.951003 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:36.950925    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6-metrics-server-audit-profiles\") pod \"metrics-server-7b6db58fc-994xl\" (UID: \"0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6\") " pod="openshift-monitoring/metrics-server-7b6db58fc-994xl"
Nov 01 05:03:36.951003 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:36.950951    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55zt7\" (UniqueName: \"kubernetes.io/projected/0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6-kube-api-access-55zt7\") pod \"metrics-server-7b6db58fc-994xl\" (UID: \"0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6\") " pod="openshift-monitoring/metrics-server-7b6db58fc-994xl"
Nov 01 05:03:36.951003 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:36.950980    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7b6db58fc-994xl\" (UID: \"0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6\") " pod="openshift-monitoring/metrics-server-7b6db58fc-994xl"
Nov 01 05:03:37.052560 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:37.051731    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6-metrics-server-audit-profiles\") pod \"metrics-server-7b6db58fc-994xl\" (UID: \"0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6\") " pod="openshift-monitoring/metrics-server-7b6db58fc-994xl"
Nov 01 05:03:37.052560 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:37.051769    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55zt7\" (UniqueName: \"kubernetes.io/projected/0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6-kube-api-access-55zt7\") pod \"metrics-server-7b6db58fc-994xl\" (UID: \"0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6\") " pod="openshift-monitoring/metrics-server-7b6db58fc-994xl"
Nov 01 05:03:37.052560 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:37.051798    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7b6db58fc-994xl\" (UID: \"0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6\") " pod="openshift-monitoring/metrics-server-7b6db58fc-994xl"
Nov 01 05:03:37.052560 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:37.051838    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6-secret-metrics-server-tls\") pod \"metrics-server-7b6db58fc-994xl\" (UID: \"0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6\") " pod="openshift-monitoring/metrics-server-7b6db58fc-994xl"
Nov 01 05:03:37.052560 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:37.051873    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6-client-ca-bundle\") pod \"metrics-server-7b6db58fc-994xl\" (UID: \"0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6\") " pod="openshift-monitoring/metrics-server-7b6db58fc-994xl"
Nov 01 05:03:37.052560 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:37.051924    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6-secret-metrics-server-client-certs\") pod \"metrics-server-7b6db58fc-994xl\" (UID: \"0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6\") " pod="openshift-monitoring/metrics-server-7b6db58fc-994xl"
Nov 01 05:03:37.052560 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:37.051962    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6-audit-log\") pod \"metrics-server-7b6db58fc-994xl\" (UID: \"0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6\") " pod="openshift-monitoring/metrics-server-7b6db58fc-994xl"
Nov 01 05:03:37.053324 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:37.053090    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6-metrics-server-audit-profiles\") pod \"metrics-server-7b6db58fc-994xl\" (UID: \"0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6\") " pod="openshift-monitoring/metrics-server-7b6db58fc-994xl"
Nov 01 05:03:37.053324 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:37.053266    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6-audit-log\") pod \"metrics-server-7b6db58fc-994xl\" (UID: \"0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6\") " pod="openshift-monitoring/metrics-server-7b6db58fc-994xl"
Nov 01 05:03:37.053732 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:37.053703    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7b6db58fc-994xl\" (UID: \"0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6\") " pod="openshift-monitoring/metrics-server-7b6db58fc-994xl"
Nov 01 05:03:37.055063 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:37.055042    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6-client-ca-bundle\") pod \"metrics-server-7b6db58fc-994xl\" (UID: \"0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6\") " pod="openshift-monitoring/metrics-server-7b6db58fc-994xl"
Nov 01 05:03:37.056483 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:37.056320    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6-secret-metrics-server-tls\") pod \"metrics-server-7b6db58fc-994xl\" (UID: \"0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6\") " pod="openshift-monitoring/metrics-server-7b6db58fc-994xl"
Nov 01 05:03:37.056865 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:37.056817    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6-secret-metrics-server-client-certs\") pod \"metrics-server-7b6db58fc-994xl\" (UID: \"0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6\") " pod="openshift-monitoring/metrics-server-7b6db58fc-994xl"
Nov 01 05:03:37.115222 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:37.115189    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55zt7\" (UniqueName: \"kubernetes.io/projected/0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6-kube-api-access-55zt7\") pod \"metrics-server-7b6db58fc-994xl\" (UID: \"0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6\") " pod="openshift-monitoring/metrics-server-7b6db58fc-994xl"
Nov 01 05:03:37.172067 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:37.172031    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7b6db58fc-994xl"
Nov 01 05:03:37.639418 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:37.639345    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7b6db58fc-994xl"]
Nov 01 05:03:37.644148 ip-10-0-129-92 kubenswrapper[2571]: W1101 05:03:37.644108    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ddf3d10_7212_4e89_b7a8_e9545a1bcaf6.slice/crio-1d7634964deac3f00ec41b31563aff4f6799d2d66d08ad1fcfed920814d877a6 WatchSource:0}: Error finding container 1d7634964deac3f00ec41b31563aff4f6799d2d66d08ad1fcfed920814d877a6: Status 404 returned error can't find the container with id 1d7634964deac3f00ec41b31563aff4f6799d2d66d08ad1fcfed920814d877a6
Nov 01 05:03:37.699352 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:37.699313    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-267rv" event={"ID":"4321e0ae-e339-40f2-875a-941b1183e766","Type":"ContainerStarted","Data":"d7311216be907b8643637ebc5a22a760a1ff74973120631412401373bca53967"}
Nov 01 05:03:37.701527 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:37.701485    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7b6db58fc-994xl" event={"ID":"0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6","Type":"ContainerStarted","Data":"1d7634964deac3f00ec41b31563aff4f6799d2d66d08ad1fcfed920814d877a6"}
Nov 01 05:03:37.991137 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:37.991090    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:38.180599 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:38.180411    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:38.560866 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:38.558134    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-sr4n8"
Nov 01 05:03:38.682274 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:38.682142    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-267rv" podStartSLOduration=67.243975445 podStartE2EDuration="1m9.682125568s" totalImagesPullingTime="2.438150123s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:02:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:03:37.81905202 +0000 UTC m=+70.019949389" watchObservedRunningTime="2026-11-01 05:03:38.682125568 +0000 UTC m=+70.883022919"
Nov 01 05:03:38.761327 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:38.761007    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:38.942197 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:38.942165    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:39.716588 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:39.716550    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77ddfd4d86-btjkv" event={"ID":"80c10aac-5175-46d6-a18b-109169a2461b","Type":"ContainerStarted","Data":"ef282cf77be9bc979e4022e0d17f30d069582387c7c295b2eaa396b0366bde92"}
Nov 01 05:03:39.716824 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:39.716684    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77ddfd4d86-btjkv" event={"ID":"80c10aac-5175-46d6-a18b-109169a2461b","Type":"ContainerStarted","Data":"bb49f4ffdf27dda1c2987b847cb8d54bfc7464e2aead04aacee482cebec72c8d"}
Nov 01 05:03:39.716824 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:39.716731    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77ddfd4d86-btjkv" event={"ID":"80c10aac-5175-46d6-a18b-109169a2461b","Type":"ContainerStarted","Data":"b1c9228d6985d143e29f445f5c157d31dd8955c81da00274508e10b296f3dd22"}
Nov 01 05:03:40.722840 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:40.722804    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7b6db58fc-994xl" event={"ID":"0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6","Type":"ContainerStarted","Data":"f7e105766b08f69278245f46f0ee788f4b5b7cc746053998161f6bb41dcc6741"}
Nov 01 05:03:40.847038 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:40.846976    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7b6db58fc-994xl" podStartSLOduration=2.645744241 podStartE2EDuration="4.846959196s" totalImagesPullingTime="2.201214955s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:03:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:03:40.845088919 +0000 UTC m=+73.045986282" watchObservedRunningTime="2026-11-01 05:03:40.846959196 +0000 UTC m=+73.047856546"
Nov 01 05:03:41.018347 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:41.018311    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:41.359619 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:41.359551    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:41.555385 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:41.555354    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:41.727840 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:41.727801    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9pcxf" event={"ID":"e856df5a-5c5e-4421-ae7d-80a760bc72e2","Type":"ContainerStarted","Data":"b6d30bead7df8ab7d81c9eb9f0098428fea48da9c860f3cc98e3bde2c686f43d"}
Nov 01 05:03:41.730855 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:41.730827    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77ddfd4d86-btjkv" event={"ID":"80c10aac-5175-46d6-a18b-109169a2461b","Type":"ContainerStarted","Data":"92e5b0c2f45ab829f0020479e42d599de06ab5269f88397218ca0787b268392c"}
Nov 01 05:03:41.730999 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:41.730886    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77ddfd4d86-btjkv" event={"ID":"80c10aac-5175-46d6-a18b-109169a2461b","Type":"ContainerStarted","Data":"0983971a3f5aaffdfe7b9b9d7fcd2357d57364db4dceebe0fdeecffe5c2101bc"}
Nov 01 05:03:41.730999 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:41.730914    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77ddfd4d86-btjkv" event={"ID":"80c10aac-5175-46d6-a18b-109169a2461b","Type":"ContainerStarted","Data":"037cf48bdca7f6df1f7395b6649eb93566ad83fd31530036e207bc714ab1a3bb"}
Nov 01 05:03:41.844541 ip-10-0-129-92 kubenswrapper[2571]: I1101 05:03:41.844495    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-9pcxf" podStartSLOduration=65.256386504 podStartE2EDuration="1m12.844479034s" totalImagesPullingTime="7.58809253s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 05:02:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:03:41.843377663 +0000 UTC m=+74.044275016" watchObservedRunningTime="2026-11-01 05:03:41.844479034 +0000 UTC m=+74.045376382"
Nov 01 05:03:41.867918 ip-10-0-129-92 systemd[1]: Stopping Kubernetes Kubelet...
Nov 01 05:03:41.884931 ip-10-0-129-92 systemd[1]: kubelet.service: Deactivated successfully.
Nov 01 05:03:41.885091 ip-10-0-129-92 systemd[1]: Stopped Kubernetes Kubelet.
Nov 01 05:03:41.885139 ip-10-0-129-92 systemd[1]: kubelet.service: Consumed 4.143s CPU time.
Nov 01 05:03:41.896962 ip-10-0-129-92 systemd[1]: Starting Kubernetes Kubelet...
Nov 01 05:03:41.973778 ip-10-0-129-92 kubenswrapper[5689]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 05:03:41.973778 ip-10-0-129-92 kubenswrapper[5689]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 01 05:03:41.973778 ip-10-0-129-92 kubenswrapper[5689]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 05:03:41.973778 ip-10-0-129-92 kubenswrapper[5689]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 05:03:41.975900 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.975818    5689 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 05:03:41.975900 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.975898    5689 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 05:03:41.975900 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.975902    5689 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 05:03:41.975900 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.975905    5689 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 05:03:41.976072 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.975909    5689 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 05:03:41.976072 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.975912    5689 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 05:03:41.976072 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.975917    5689 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 05:03:41.976072 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.975921    5689 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 05:03:41.976072 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.975924    5689 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 05:03:41.976072 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.975927    5689 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 05:03:41.976072 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.975930    5689 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 05:03:41.976072 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.975932    5689 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 05:03:41.976072 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.975935    5689 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 05:03:41.976072 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.975937    5689 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 05:03:41.976072 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.975947    5689 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 05:03:41.976072 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.975950    5689 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 05:03:41.976072 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.975953    5689 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 05:03:41.976072 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.975955    5689 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 05:03:41.976072 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.975958    5689 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 05:03:41.976072 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.975960    5689 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 05:03:41.976072 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.975963    5689 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 05:03:41.976072 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.975965    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 05:03:41.976072 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.975968    5689 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 05:03:41.976616 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.975970    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 05:03:41.976616 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.975973    5689 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 05:03:41.976616 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.975975    5689 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 05:03:41.976616 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.975978    5689 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 05:03:41.976616 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.975980    5689 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 05:03:41.976616 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.975983    5689 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 05:03:41.976616 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.975986    5689 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 05:03:41.976616 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.975989    5689 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 05:03:41.976616 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.975991    5689 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 05:03:41.976616 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.975994    5689 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 05:03:41.976616 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.975997    5689 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 05:03:41.976616 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.976000    5689 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 05:03:41.976616 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.976003    5689 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 05:03:41.976616 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.976005    5689 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 05:03:41.976616 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.976008    5689 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 05:03:41.976616 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.976010    5689 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 05:03:41.976616 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.976013    5689 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 05:03:41.976616 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.976016    5689 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 05:03:41.976616 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.976018    5689 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 05:03:41.976616 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.976021    5689 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 05:03:41.977180 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.976023    5689 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 05:03:41.977180 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.976026    5689 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 05:03:41.977180 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.976028    5689 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 05:03:41.977180 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.976031    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 05:03:41.977180 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.976034    5689 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 05:03:41.977180 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.976037    5689 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 05:03:41.977180 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.976039    5689 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 05:03:41.977180 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.976042    5689 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 05:03:41.977180 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.976044    5689 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 05:03:41.977180 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.976047    5689 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 05:03:41.977180 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.976049    5689 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 05:03:41.977180 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.976052    5689 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 05:03:41.977180 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.976055    5689 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 05:03:41.977180 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.976057    5689 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 05:03:41.977180 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.976060    5689 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 05:03:41.977180 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.976062    5689 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 05:03:41.977180 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.976065    5689 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 05:03:41.977180 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.976067    5689 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 05:03:41.977180 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.976072    5689 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 05:03:41.977180 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.976074    5689 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 05:03:41.977697 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.976077    5689 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 05:03:41.977697 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.976080    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 05:03:41.977697 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.976082    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 05:03:41.977697 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.976086    5689 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 05:03:41.977697 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.976088    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 05:03:41.977697 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.976091    5689 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 05:03:41.977697 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.976093    5689 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 05:03:41.977697 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.976096    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 05:03:41.977697 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.976098    5689 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 05:03:41.977697 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.976102    5689 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 05:03:41.977697 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.976104    5689 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 05:03:41.977697 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.976107    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 05:03:41.977697 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.976109    5689 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 05:03:41.977697 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.976113    5689 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 05:03:41.977697 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.976117    5689 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 05:03:41.977697 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.976119    5689 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 05:03:41.977697 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.976122    5689 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 05:03:41.977697 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.976125    5689 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 05:03:41.978196 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.976128    5689 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 05:03:41.978196 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.976131    5689 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 05:03:41.978196 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.976133    5689 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 05:03:41.978196 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.976136    5689 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 05:03:41.978196 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.976138    5689 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 05:03:41.978196 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.976140    5689 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 05:03:41.978196 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.976143    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 05:03:41.978196 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.976146    5689 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 05:03:41.978196 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.976148    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 05:03:41.978196 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.976151    5689 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 05:03:41.978196 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.976153    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 05:03:41.978196 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.976156    5689 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 05:03:41.978196 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.976158    5689 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 05:03:41.978196 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.976161    5689 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 05:03:41.978196 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.976163    5689 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 05:03:41.978196 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.976166    5689 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 05:03:41.978196 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.976168    5689 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 05:03:41.978196 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.976172    5689 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 05:03:41.978196 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.976174    5689 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 05:03:41.978196 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.976177    5689 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 05:03:41.978720 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.976180    5689 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 05:03:41.978720 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.976182    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 05:03:41.978720 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.976185    5689 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 05:03:41.978720 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.976187    5689 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 05:03:41.978720 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.976191    5689 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 05:03:41.978720 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977000    5689 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 05:03:41.978720 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977009    5689 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 05:03:41.978720 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977013    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 05:03:41.978720 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977016    5689 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 05:03:41.978720 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977019    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 05:03:41.978720 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977022    5689 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 05:03:41.978720 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977024    5689 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 05:03:41.978720 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977027    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 05:03:41.978720 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977030    5689 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 05:03:41.978720 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977033    5689 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 05:03:41.978720 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977035    5689 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 05:03:41.978720 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977038    5689 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 05:03:41.978720 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977040    5689 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 05:03:41.978720 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977043    5689 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 05:03:41.979201 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977045    5689 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 05:03:41.979201 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977048    5689 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 05:03:41.979201 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977050    5689 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 05:03:41.979201 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977053    5689 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 05:03:41.979201 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977056    5689 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 05:03:41.979201 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977058    5689 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 05:03:41.979201 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977062    5689 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 05:03:41.979201 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977064    5689 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 05:03:41.979201 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977067    5689 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 05:03:41.979201 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977070    5689 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 05:03:41.979201 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977072    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 05:03:41.979201 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977075    5689 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 05:03:41.979201 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977077    5689 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 05:03:41.979201 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977080    5689 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 05:03:41.979201 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977083    5689 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 05:03:41.979201 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977085    5689 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 05:03:41.979201 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977088    5689 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 05:03:41.979201 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977090    5689 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 05:03:41.979201 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977093    5689 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 05:03:41.979201 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977096    5689 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 05:03:41.979721 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977098    5689 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 05:03:41.979721 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977101    5689 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 05:03:41.979721 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977103    5689 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 05:03:41.979721 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977106    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 05:03:41.979721 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977108    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 05:03:41.979721 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977111    5689 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 05:03:41.979721 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977114    5689 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 05:03:41.979721 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977117    5689 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 05:03:41.979721 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977119    5689 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 05:03:41.979721 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977122    5689 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 05:03:41.979721 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977125    5689 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 05:03:41.979721 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977128    5689 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 05:03:41.979721 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977132    5689 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 05:03:41.979721 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977135    5689 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 05:03:41.979721 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977138    5689 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 05:03:41.979721 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977141    5689 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 05:03:41.979721 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977143    5689 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 05:03:41.979721 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977146    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 05:03:41.979721 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977148    5689 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 05:03:41.980195 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977151    5689 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 05:03:41.980195 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977154    5689 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 05:03:41.980195 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977156    5689 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 05:03:41.980195 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977159    5689 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 05:03:41.980195 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977161    5689 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 05:03:41.980195 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977164    5689 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 05:03:41.980195 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977167    5689 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 05:03:41.980195 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977170    5689 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 05:03:41.980195 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977174    5689 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 05:03:41.980195 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977178    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 05:03:41.980195 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977181    5689 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 05:03:41.980195 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977184    5689 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 05:03:41.980195 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977187    5689 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 05:03:41.980195 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977190    5689 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 05:03:41.980195 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977193    5689 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 05:03:41.980195 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977196    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 05:03:41.980195 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977199    5689 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 05:03:41.980195 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977201    5689 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 05:03:41.980195 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977204    5689 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 05:03:41.980195 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977206    5689 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 05:03:41.980707 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977209    5689 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 05:03:41.980707 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977212    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 05:03:41.980707 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977215    5689 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 05:03:41.980707 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977217    5689 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 05:03:41.980707 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977220    5689 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 05:03:41.980707 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977222    5689 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 05:03:41.980707 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977225    5689 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 05:03:41.980707 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977227    5689 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 05:03:41.980707 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977230    5689 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 05:03:41.980707 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977236    5689 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 05:03:41.980707 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977238    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 05:03:41.980707 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977241    5689 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 05:03:41.980707 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977244    5689 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 05:03:41.980707 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977246    5689 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 05:03:41.980707 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977249    5689 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 05:03:41.980707 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977252    5689 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 05:03:41.980707 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977254    5689 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 05:03:41.980707 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977257    5689 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 05:03:41.980707 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977260    5689 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 05:03:41.981177 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977262    5689 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 05:03:41.981177 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977265    5689 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 05:03:41.981177 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977268    5689 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 05:03:41.981177 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977270    5689 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 05:03:41.981177 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977274    5689 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 05:03:41.981177 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977276    5689 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 05:03:41.981177 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977279    5689 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 05:03:41.981177 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977281    5689 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 05:03:41.981177 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977284    5689 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 05:03:41.981177 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977287    5689 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 05:03:41.981177 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977289    5689 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 05:03:41.981177 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977292    5689 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 05:03:41.981177 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977294    5689 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 05:03:41.981177 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977297    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 05:03:41.981177 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977819    5689 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 05:03:41.981177 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977826    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 05:03:41.981177 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977829    5689 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 05:03:41.981177 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977832    5689 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 05:03:41.981177 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977835    5689 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 05:03:41.981177 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977838    5689 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 05:03:41.981700 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977841    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 05:03:41.981700 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977844    5689 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 05:03:41.981700 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977849    5689 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 05:03:41.981700 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977852    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 05:03:41.981700 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977855    5689 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 05:03:41.981700 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977858    5689 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 05:03:41.981700 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977861    5689 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 05:03:41.981700 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977864    5689 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 05:03:41.981700 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977866    5689 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 05:03:41.981700 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977869    5689 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 05:03:41.981700 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977880    5689 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 05:03:41.981700 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977883    5689 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 05:03:41.981700 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977886    5689 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 05:03:41.981700 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977888    5689 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 05:03:41.981700 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977892    5689 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 05:03:41.981700 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977897    5689 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 05:03:41.981700 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977900    5689 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 05:03:41.981700 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977903    5689 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 05:03:41.981700 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977906    5689 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 05:03:41.982225 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977908    5689 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 05:03:41.982225 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977911    5689 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 05:03:41.982225 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977914    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 05:03:41.982225 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977917    5689 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 05:03:41.982225 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977919    5689 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 05:03:41.982225 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977922    5689 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 05:03:41.982225 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977925    5689 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 05:03:41.982225 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977928    5689 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 05:03:41.982225 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977930    5689 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 05:03:41.982225 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977933    5689 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 05:03:41.982225 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977935    5689 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 05:03:41.982225 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977938    5689 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 05:03:41.982225 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977941    5689 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 05:03:41.982225 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977943    5689 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 05:03:41.982225 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977946    5689 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 05:03:41.982225 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977950    5689 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 05:03:41.982225 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977953    5689 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 05:03:41.982225 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977955    5689 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 05:03:41.982225 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977958    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 05:03:41.982709 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977961    5689 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 05:03:41.982709 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977963    5689 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 05:03:41.982709 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977966    5689 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 05:03:41.982709 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977968    5689 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 05:03:41.982709 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977971    5689 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 05:03:41.982709 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977974    5689 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 05:03:41.982709 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977976    5689 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 05:03:41.982709 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977979    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 05:03:41.982709 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977981    5689 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 05:03:41.982709 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977984    5689 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 05:03:41.982709 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977986    5689 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 05:03:41.982709 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977989    5689 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 05:03:41.982709 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977992    5689 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 05:03:41.982709 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977995    5689 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 05:03:41.982709 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.977997    5689 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 05:03:41.982709 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.978001    5689 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 05:03:41.982709 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.978004    5689 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 05:03:41.982709 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.978006    5689 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 05:03:41.982709 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.978009    5689 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 05:03:41.983164 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.978011    5689 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 05:03:41.983164 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.978014    5689 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 05:03:41.983164 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.978017    5689 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 05:03:41.983164 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.978020    5689 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 05:03:41.983164 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.978023    5689 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 05:03:41.983164 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.978025    5689 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 05:03:41.983164 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.978028    5689 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 05:03:41.983164 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.978030    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 05:03:41.983164 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.978033    5689 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 05:03:41.983164 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.978036    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 05:03:41.983164 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.978039    5689 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 05:03:41.983164 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.978043    5689 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 05:03:41.983164 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.978046    5689 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 05:03:41.983164 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.978049    5689 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 05:03:41.983164 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.978052    5689 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 05:03:41.983164 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.978055    5689 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 05:03:41.983164 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.978057    5689 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 05:03:41.983164 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.978060    5689 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 05:03:41.983164 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.978063    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 05:03:41.983164 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.978065    5689 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 05:03:41.983663 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.978068    5689 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 05:03:41.983663 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.978071    5689 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 05:03:41.983663 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.978073    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 05:03:41.983663 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.978076    5689 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 05:03:41.983663 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.978079    5689 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 05:03:41.983663 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.978081    5689 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 05:03:41.983663 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.978087    5689 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 05:03:41.983663 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.978090    5689 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 05:03:41.983663 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.978095    5689 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 05:03:41.983663 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.978097    5689 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 05:03:41.983663 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.978100    5689 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 05:03:41.983663 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.978103    5689 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 05:03:41.983663 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.978105    5689 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 05:03:41.983663 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.978108    5689 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 05:03:41.983663 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.978110    5689 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 05:03:41.983663 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.978113    5689 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 05:03:41.983663 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.978116    5689 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 05:03:41.983663 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.978118    5689 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 05:03:41.983663 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.978121    5689 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 05:03:41.983663 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.978124    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 05:03:41.984163 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.978126    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 05:03:41.984163 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.978131    5689 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 05:03:41.984163 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.978133    5689 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 05:03:41.984163 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978225    5689 flags.go:64] FLAG: --address="0.0.0.0"
Nov 01 05:03:41.984163 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978232    5689 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 01 05:03:41.984163 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978244    5689 flags.go:64] FLAG: --anonymous-auth="true"
Nov 01 05:03:41.984163 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978249    5689 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 01 05:03:41.984163 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978254    5689 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 01 05:03:41.984163 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978258    5689 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 01 05:03:41.984163 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978262    5689 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 01 05:03:41.984163 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978267    5689 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 01 05:03:41.984163 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978270    5689 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 01 05:03:41.984163 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978273    5689 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 01 05:03:41.984163 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978276    5689 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 01 05:03:41.984163 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978279    5689 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 01 05:03:41.984163 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978283    5689 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 01 05:03:41.984163 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978286    5689 flags.go:64] FLAG: --cgroup-root=""
Nov 01 05:03:41.984163 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978288    5689 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 01 05:03:41.984163 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978291    5689 flags.go:64] FLAG: --client-ca-file=""
Nov 01 05:03:41.984163 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978294    5689 flags.go:64] FLAG: --cloud-provider="external"
Nov 01 05:03:41.984163 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978298    5689 flags.go:64] FLAG: --cluster-dns="[]"
Nov 01 05:03:41.984163 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978304    5689 flags.go:64] FLAG: --cluster-domain=""
Nov 01 05:03:41.984163 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978307    5689 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 01 05:03:41.984163 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978310    5689 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 01 05:03:41.985042 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978313    5689 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 01 05:03:41.985042 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978317    5689 flags.go:64] FLAG: --container-log-max-files="5"
Nov 01 05:03:41.985042 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978321    5689 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 01 05:03:41.985042 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978324    5689 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 01 05:03:41.985042 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978333    5689 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 01 05:03:41.985042 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978336    5689 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 01 05:03:41.985042 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978339    5689 flags.go:64] FLAG: --contention-profiling="false"
Nov 01 05:03:41.985042 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978342    5689 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 01 05:03:41.985042 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978345    5689 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 01 05:03:41.985042 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978349    5689 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 01 05:03:41.985042 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978352    5689 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 01 05:03:41.985042 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978357    5689 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 01 05:03:41.985042 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978360    5689 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 01 05:03:41.985042 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978363    5689 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 01 05:03:41.985042 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978366    5689 flags.go:64] FLAG: --enable-load-reader="false"
Nov 01 05:03:41.985042 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978369    5689 flags.go:64] FLAG: --enable-server="true"
Nov 01 05:03:41.985042 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978371    5689 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 01 05:03:41.985042 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978379    5689 flags.go:64] FLAG: --event-burst="100"
Nov 01 05:03:41.985042 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978382    5689 flags.go:64] FLAG: --event-qps="50"
Nov 01 05:03:41.985042 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978385    5689 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 01 05:03:41.985042 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978388    5689 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 01 05:03:41.985042 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978391    5689 flags.go:64] FLAG: --eviction-hard=""
Nov 01 05:03:41.985042 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978395    5689 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 01 05:03:41.985042 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978398    5689 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 01 05:03:41.985862 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978402    5689 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 01 05:03:41.985862 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978405    5689 flags.go:64] FLAG: --eviction-soft=""
Nov 01 05:03:41.985862 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978407    5689 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 01 05:03:41.985862 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978411    5689 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 01 05:03:41.985862 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978414    5689 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 01 05:03:41.985862 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978417    5689 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 01 05:03:41.985862 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978420    5689 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 01 05:03:41.985862 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978423    5689 flags.go:64] FLAG: --fail-swap-on="true"
Nov 01 05:03:41.985862 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978426    5689 flags.go:64] FLAG: --feature-gates=""
Nov 01 05:03:41.985862 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978430    5689 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 01 05:03:41.985862 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978433    5689 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 01 05:03:41.985862 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978436    5689 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 01 05:03:41.985862 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978440    5689 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 01 05:03:41.985862 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978466    5689 flags.go:64] FLAG: --healthz-port="10248"
Nov 01 05:03:41.985862 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978473    5689 flags.go:64] FLAG: --help="false"
Nov 01 05:03:41.985862 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978486    5689 flags.go:64] FLAG: --hostname-override="ip-10-0-129-92.us-west-1.compute.internal"
Nov 01 05:03:41.985862 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978492    5689 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 01 05:03:41.985862 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978497    5689 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 01 05:03:41.985862 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978500    5689 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 01 05:03:41.985862 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978504    5689 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 01 05:03:41.985862 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978507    5689 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 01 05:03:41.985862 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978510    5689 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 01 05:03:41.985862 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978513    5689 flags.go:64] FLAG: --image-service-endpoint=""
Nov 01 05:03:41.986426 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978515    5689 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 01 05:03:41.986426 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978519    5689 flags.go:64] FLAG: --kube-api-burst="100"
Nov 01 05:03:41.986426 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978522    5689 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 01 05:03:41.986426 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978526    5689 flags.go:64] FLAG: --kube-api-qps="50"
Nov 01 05:03:41.986426 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978528    5689 flags.go:64] FLAG: --kube-reserved=""
Nov 01 05:03:41.986426 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978531    5689 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 01 05:03:41.986426 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978534    5689 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 01 05:03:41.986426 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978537    5689 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 01 05:03:41.986426 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978540    5689 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 01 05:03:41.986426 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978543    5689 flags.go:64] FLAG: --lock-file=""
Nov 01 05:03:41.986426 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978546    5689 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 01 05:03:41.986426 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978549    5689 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 01 05:03:41.986426 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978552    5689 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 01 05:03:41.986426 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978557    5689 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 01 05:03:41.986426 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978560    5689 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 01 05:03:41.986426 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978563    5689 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 01 05:03:41.986426 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978566    5689 flags.go:64] FLAG: --logging-format="text"
Nov 01 05:03:41.986426 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978569    5689 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 01 05:03:41.986426 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978572    5689 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 01 05:03:41.986426 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978575    5689 flags.go:64] FLAG: --manifest-url=""
Nov 01 05:03:41.986426 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978577    5689 flags.go:64] FLAG: --manifest-url-header=""
Nov 01 05:03:41.986426 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978581    5689 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 01 05:03:41.986426 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978584    5689 flags.go:64] FLAG: --max-open-files="1000000"
Nov 01 05:03:41.986426 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978596    5689 flags.go:64] FLAG: --max-pods="110"
Nov 01 05:03:41.986426 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978600    5689 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 01 05:03:41.987050 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978602    5689 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 01 05:03:41.987050 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978606    5689 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 01 05:03:41.987050 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978609    5689 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 01 05:03:41.987050 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978612    5689 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 01 05:03:41.987050 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978615    5689 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 01 05:03:41.987050 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978619    5689 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 01 05:03:41.987050 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978626    5689 flags.go:64] FLAG: --node-status-max-images="50"
Nov 01 05:03:41.987050 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978629    5689 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 01 05:03:41.987050 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978632    5689 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 01 05:03:41.987050 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978634    5689 flags.go:64] FLAG: --pod-cidr=""
Nov 01 05:03:41.987050 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978637    5689 flags.go:64] FLAG: --pod-manifest-path=""
Nov 01 05:03:41.987050 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978640    5689 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 01 05:03:41.987050 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978643    5689 flags.go:64] FLAG: --pods-per-core="0"
Nov 01 05:03:41.987050 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978646    5689 flags.go:64] FLAG: --port="10250"
Nov 01 05:03:41.987050 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978648    5689 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 01 05:03:41.987050 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978651    5689 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0fdeb5f686df6389c"
Nov 01 05:03:41.987050 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978654    5689 flags.go:64] FLAG: --qos-reserved=""
Nov 01 05:03:41.987050 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978659    5689 flags.go:64] FLAG: --read-only-port="10255"
Nov 01 05:03:41.987050 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978662    5689 flags.go:64] FLAG: --register-node="true"
Nov 01 05:03:41.987050 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978665    5689 flags.go:64] FLAG: --register-with-taints=""
Nov 01 05:03:41.987050 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978669    5689 flags.go:64] FLAG: --registry-burst="10"
Nov 01 05:03:41.987050 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978672    5689 flags.go:64] FLAG: --registry-qps="5"
Nov 01 05:03:41.987050 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978675    5689 flags.go:64] FLAG: --reserved-cpus=""
Nov 01 05:03:41.987050 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978677    5689 flags.go:64] FLAG: --reserved-memory=""
Nov 01 05:03:41.987050 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978681    5689 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 01 05:03:41.987050 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978684    5689 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 01 05:03:41.987714 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978687    5689 flags.go:64] FLAG: --rotate-certificates="false"
Nov 01 05:03:41.987714 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978690    5689 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 01 05:03:41.987714 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978693    5689 flags.go:64] FLAG: --runonce="false"
Nov 01 05:03:41.987714 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978697    5689 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 01 05:03:41.987714 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978700    5689 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 01 05:03:41.987714 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978708    5689 flags.go:64] FLAG: --seccomp-default="false"
Nov 01 05:03:41.987714 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978711    5689 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 01 05:03:41.987714 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978715    5689 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 01 05:03:41.987714 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978718    5689 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 01 05:03:41.987714 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978722    5689 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 01 05:03:41.987714 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978725    5689 flags.go:64] FLAG: --storage-driver-password="root"
Nov 01 05:03:41.987714 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978728    5689 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 01 05:03:41.987714 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978731    5689 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 01 05:03:41.987714 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978734    5689 flags.go:64] FLAG: --storage-driver-user="root"
Nov 01 05:03:41.987714 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978736    5689 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 01 05:03:41.987714 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978740    5689 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 01 05:03:41.987714 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978742    5689 flags.go:64] FLAG: --system-cgroups=""
Nov 01 05:03:41.987714 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978745    5689 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 01 05:03:41.987714 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978754    5689 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 01 05:03:41.987714 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978757    5689 flags.go:64] FLAG: --tls-cert-file=""
Nov 01 05:03:41.987714 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978760    5689 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 01 05:03:41.987714 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978766    5689 flags.go:64] FLAG: --tls-min-version=""
Nov 01 05:03:41.987714 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978769    5689 flags.go:64] FLAG: --tls-private-key-file=""
Nov 01 05:03:41.987714 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978774    5689 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 01 05:03:41.987714 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978776    5689 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 01 05:03:41.988514 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978779    5689 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 01 05:03:41.988514 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978782    5689 flags.go:64] FLAG: --v="2"
Nov 01 05:03:41.988514 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978786    5689 flags.go:64] FLAG: --version="false"
Nov 01 05:03:41.988514 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978790    5689 flags.go:64] FLAG: --vmodule=""
Nov 01 05:03:41.988514 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978794    5689 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 01 05:03:41.988514 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.978797    5689 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 01 05:03:41.988514 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.978940    5689 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 05:03:41.988514 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.978944    5689 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 05:03:41.988514 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.978947    5689 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 05:03:41.988514 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.978950    5689 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 05:03:41.988514 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.978952    5689 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 05:03:41.988514 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.978955    5689 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 05:03:41.988514 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.978957    5689 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 05:03:41.988514 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.978963    5689 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 05:03:41.988514 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.978967    5689 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 05:03:41.988514 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.978970    5689 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 05:03:41.988514 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.978972    5689 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 05:03:41.988514 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.978975    5689 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 05:03:41.988514 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.978978    5689 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 05:03:41.988514 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.978980    5689 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 05:03:41.988514 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.978983    5689 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 05:03:41.989107 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.978985    5689 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 05:03:41.989107 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.978988    5689 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 05:03:41.989107 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.978990    5689 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 05:03:41.989107 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.978993    5689 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 05:03:41.989107 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.978995    5689 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 05:03:41.989107 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.978998    5689 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 05:03:41.989107 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979000    5689 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 05:03:41.989107 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979003    5689 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 05:03:41.989107 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979006    5689 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 05:03:41.989107 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979010    5689 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 05:03:41.989107 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979013    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 05:03:41.989107 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979016    5689 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 05:03:41.989107 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979018    5689 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 05:03:41.989107 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979021    5689 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 05:03:41.989107 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979024    5689 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 05:03:41.989107 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979026    5689 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 05:03:41.989107 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979029    5689 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 05:03:41.989107 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979031    5689 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 05:03:41.989107 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979034    5689 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 05:03:41.989665 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979037    5689 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 05:03:41.989665 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979039    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 05:03:41.989665 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979042    5689 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 05:03:41.989665 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979045    5689 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 05:03:41.989665 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979047    5689 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 05:03:41.989665 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979050    5689 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 05:03:41.989665 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979054    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 05:03:41.989665 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979056    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 05:03:41.989665 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979059    5689 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 05:03:41.989665 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979061    5689 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 05:03:41.989665 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979064    5689 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 05:03:41.989665 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979066    5689 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 05:03:41.989665 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979069    5689 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 05:03:41.989665 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979072    5689 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 05:03:41.989665 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979074    5689 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 05:03:41.989665 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979076    5689 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 05:03:41.989665 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979079    5689 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 05:03:41.989665 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979081    5689 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 05:03:41.989665 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979084    5689 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 05:03:41.989665 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979086    5689 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 05:03:41.990263 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979089    5689 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 05:03:41.990263 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979091    5689 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 05:03:41.990263 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979095    5689 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 05:03:41.990263 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979098    5689 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 05:03:41.990263 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979101    5689 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 05:03:41.990263 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979103    5689 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 05:03:41.990263 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979106    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 05:03:41.990263 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979108    5689 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 05:03:41.990263 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979111    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 05:03:41.990263 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979113    5689 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 05:03:41.990263 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979116    5689 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 05:03:41.990263 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979118    5689 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 05:03:41.990263 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979121    5689 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 05:03:41.990263 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979124    5689 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 05:03:41.990263 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979126    5689 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 05:03:41.990263 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979128    5689 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 05:03:41.990263 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979131    5689 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 05:03:41.990263 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979134    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 05:03:41.990263 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979138    5689 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 05:03:41.990263 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979146    5689 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 05:03:41.991025 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979149    5689 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 05:03:41.991025 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979151    5689 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 05:03:41.991025 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979154    5689 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 05:03:41.991025 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979157    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 05:03:41.991025 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979160    5689 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 05:03:41.991025 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979162    5689 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 05:03:41.991025 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979164    5689 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 05:03:41.991025 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979167    5689 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 05:03:41.991025 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979169    5689 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 05:03:41.991025 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979172    5689 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 05:03:41.991025 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979176    5689 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 05:03:41.991025 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979180    5689 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 05:03:41.991025 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979184    5689 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 05:03:41.991025 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979186    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 05:03:41.991025 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979191    5689 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 05:03:41.991025 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979193    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 05:03:41.991025 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979196    5689 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 05:03:41.991025 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979199    5689 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 05:03:41.991025 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979202    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 05:03:41.991659 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979205    5689 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 05:03:41.991659 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979207    5689 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 05:03:41.991659 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979210    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 05:03:41.991659 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979213    5689 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 05:03:41.991659 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979215    5689 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 05:03:41.991659 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979218    5689 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 05:03:41.991659 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979220    5689 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 05:03:41.991659 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979223    5689 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 05:03:41.991659 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979225    5689 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 05:03:41.991659 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979228    5689 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 05:03:41.991659 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979232    5689 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 05:03:41.991659 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979236    5689 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 05:03:41.991659 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979239    5689 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 05:03:41.991659 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.979271    5689 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 05:03:41.991659 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979313    5689 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 05:03:41.991659 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979317    5689 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 05:03:41.992049 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979320    5689 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 05:03:41.992049 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979323    5689 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 05:03:41.992049 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979325    5689 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 05:03:41.992049 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979328    5689 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 05:03:41.992049 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979330    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 05:03:41.992049 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979333    5689 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 05:03:41.992049 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979335    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 05:03:41.992049 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979338    5689 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 05:03:41.992049 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979341    5689 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 05:03:41.992049 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979344    5689 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 05:03:41.992049 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979346    5689 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 05:03:41.992049 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979349    5689 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 05:03:41.992049 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979353    5689 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 05:03:41.992049 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979355    5689 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 05:03:41.992049 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979358    5689 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 05:03:41.992049 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979360    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 05:03:41.992049 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979363    5689 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 05:03:41.992049 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979365    5689 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 05:03:41.992049 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979368    5689 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 05:03:41.992049 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979371    5689 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 05:03:41.992670 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979373    5689 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 05:03:41.992670 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979381    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 05:03:41.992670 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979384    5689 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 05:03:41.992670 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979387    5689 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 05:03:41.992670 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979390    5689 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 05:03:41.992670 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979392    5689 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 05:03:41.992670 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979395    5689 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 05:03:41.992670 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979397    5689 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 05:03:41.992670 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979401    5689 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 05:03:41.992670 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979405    5689 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 05:03:41.992670 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979408    5689 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 05:03:41.992670 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979412    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 05:03:41.992670 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979415    5689 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 05:03:41.992670 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979418    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 05:03:41.992670 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979421    5689 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 05:03:41.992670 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979423    5689 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 05:03:41.992670 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979425    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 05:03:41.992670 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979428    5689 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 05:03:41.992670 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979430    5689 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 05:03:41.993274 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979433    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 05:03:41.993274 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979435    5689 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 05:03:41.993274 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979438    5689 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 05:03:41.993274 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979440    5689 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 05:03:41.993274 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979443    5689 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 05:03:41.993274 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979446    5689 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 05:03:41.993274 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979463    5689 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 05:03:41.993274 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979466    5689 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 05:03:41.993274 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979470    5689 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 05:03:41.993274 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979473    5689 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 05:03:41.993274 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979477    5689 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 05:03:41.993274 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979480    5689 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 05:03:41.993274 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979482    5689 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 05:03:41.993274 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979485    5689 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 05:03:41.993274 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979488    5689 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 05:03:41.993274 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979490    5689 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 05:03:41.993274 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979493    5689 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 05:03:41.993274 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979495    5689 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 05:03:41.993274 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979498    5689 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 05:03:41.993914 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979500    5689 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 05:03:41.993914 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979503    5689 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 05:03:41.993914 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979505    5689 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 05:03:41.993914 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979508    5689 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 05:03:41.993914 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979511    5689 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 05:03:41.993914 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979513    5689 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 05:03:41.993914 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979516    5689 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 05:03:41.993914 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979519    5689 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 05:03:41.993914 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979521    5689 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 05:03:41.993914 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979524    5689 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 05:03:41.993914 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979526    5689 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 05:03:41.993914 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979528    5689 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 05:03:41.993914 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979531    5689 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 05:03:41.993914 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979533    5689 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 05:03:41.993914 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979536    5689 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 05:03:41.993914 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979539    5689 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 05:03:41.993914 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979541    5689 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 05:03:41.993914 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979544    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 05:03:41.993914 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979547    5689 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 05:03:41.993914 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979550    5689 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 05:03:41.994424 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979552    5689 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 05:03:41.994424 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979555    5689 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 05:03:41.994424 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979557    5689 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 05:03:41.994424 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979560    5689 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 05:03:41.994424 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979563    5689 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 05:03:41.994424 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979566    5689 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 05:03:41.994424 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979569    5689 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 05:03:41.994424 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979571    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 05:03:41.994424 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979574    5689 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 05:03:41.994424 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979577    5689 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 05:03:41.994424 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979579    5689 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 05:03:41.994424 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979582    5689 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 05:03:41.994424 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979584    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 05:03:41.994424 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979587    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 05:03:41.994424 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979589    5689 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 05:03:41.994424 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979591    5689 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 05:03:41.994424 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979594    5689 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 05:03:41.994424 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979596    5689 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 05:03:41.994424 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979599    5689 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 05:03:41.994900 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979602    5689 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 05:03:41.994900 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979604    5689 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 05:03:41.994900 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979607    5689 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 05:03:41.994900 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979609    5689 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 05:03:41.994900 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979612    5689 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 05:03:41.994900 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979614    5689 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 05:03:41.994900 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.979617    5689 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 05:03:41.994900 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.983802    5689 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 01 05:03:41.994900 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.983818    5689 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 01 05:03:41.994900 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.983835    5689 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 01 05:03:41.994900 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.983841    5689 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 01 05:03:41.994900 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.983901    5689 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 05:03:41.994900 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.983906    5689 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 05:03:41.994900 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.983910    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 05:03:41.994900 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.983912    5689 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 05:03:41.994900 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.983916    5689 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 05:03:41.994900 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.983918    5689 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 05:03:41.994900 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.983921    5689 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 05:03:41.994900 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.983924    5689 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 05:03:41.994900 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.983926    5689 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 05:03:41.995375 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.983929    5689 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 05:03:41.995375 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.983932    5689 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 05:03:41.995375 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.983935    5689 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 05:03:41.995375 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.983937    5689 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 05:03:41.995375 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.983940    5689 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 05:03:41.995375 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.983942    5689 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 05:03:41.995375 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.983945    5689 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 05:03:41.995375 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.983948    5689 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 05:03:41.995375 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.983950    5689 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 05:03:41.995375 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.983953    5689 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 05:03:41.995375 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.983956    5689 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 05:03:41.995375 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.983959    5689 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 05:03:41.995375 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.983961    5689 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 05:03:41.995375 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.983964    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 05:03:41.995375 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.983966    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 05:03:41.995375 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.983969    5689 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 05:03:41.995375 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.983972    5689 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 05:03:41.995375 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.983974    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 05:03:41.995375 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.983977    5689 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 05:03:41.995375 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.983985    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 05:03:41.995878 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.983995    5689 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 05:03:41.995878 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.983998    5689 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 05:03:41.995878 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.984001    5689 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 05:03:41.995878 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.984004    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 05:03:41.995878 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.984006    5689 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 05:03:41.995878 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.984009    5689 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 05:03:41.995878 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.984012    5689 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 05:03:41.995878 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.984014    5689 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 05:03:41.995878 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.984017    5689 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 05:03:41.995878 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.984019    5689 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 05:03:41.995878 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.984022    5689 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 05:03:41.995878 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.984025    5689 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 05:03:41.995878 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.984029    5689 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 05:03:41.995878 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.984032    5689 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 05:03:41.995878 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.984034    5689 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 05:03:41.995878 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.984037    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 05:03:41.995878 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.984039    5689 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 05:03:41.995878 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.984042    5689 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 05:03:41.995878 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.984044    5689 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 05:03:41.996329 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.984047    5689 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 05:03:41.996329 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.984050    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 05:03:41.996329 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.984052    5689 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 05:03:41.996329 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.984055    5689 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 05:03:41.996329 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.984058    5689 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 05:03:41.996329 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.984061    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 05:03:41.996329 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.984064    5689 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 05:03:41.996329 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.984066    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 05:03:41.996329 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.984068    5689 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 05:03:41.996329 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.984071    5689 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 05:03:41.996329 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.984073    5689 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 05:03:41.996329 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.984076    5689 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 05:03:41.996329 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.984078    5689 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 05:03:41.996329 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.984080    5689 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 05:03:41.996329 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.984088    5689 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 05:03:41.996329 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.984091    5689 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 05:03:41.996329 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.984094    5689 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 05:03:41.996329 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.984096    5689 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 05:03:41.996329 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.984098    5689 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 05:03:41.996329 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.984101    5689 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 05:03:41.996936 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.984104    5689 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 05:03:41.996936 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.984106    5689 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 05:03:41.996936 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.984108    5689 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 05:03:41.996936 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.984111    5689 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 05:03:41.996936 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.984113    5689 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 05:03:41.996936 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.984116    5689 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 05:03:41.996936 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.984119    5689 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 05:03:41.996936 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.984123    5689 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 05:03:41.996936 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.984126    5689 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 05:03:41.996936 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.984128    5689 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 05:03:41.996936 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.984131    5689 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 05:03:41.996936 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.984133    5689 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 05:03:41.996936 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.984136    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 05:03:41.996936 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.984139    5689 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 05:03:41.996936 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.984141    5689 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 05:03:41.996936 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.984144    5689 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 05:03:41.996936 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.984146    5689 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 05:03:41.996936 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.984149    5689 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 05:03:41.996936 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.984151    5689 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 05:03:41.997418 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.984154    5689 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 05:03:41.997418 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.984156    5689 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 05:03:41.997418 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.984159    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 05:03:41.997418 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.984161    5689 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 05:03:41.997418 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.984163    5689 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 05:03:41.997418 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.984166    5689 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 05:03:41.997418 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.984169    5689 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 05:03:41.997418 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.984171    5689 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 05:03:41.997418 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.984179    5689 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 05:03:41.997418 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.984182    5689 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 05:03:41.997418 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.984185    5689 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 05:03:41.997418 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.984187    5689 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 05:03:41.997418 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.984189    5689 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 05:03:41.997418 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.984192    5689 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 05:03:41.997418 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.984194    5689 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 05:03:41.997418 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.984197    5689 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 05:03:41.997418 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.984199    5689 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 05:03:41.997418 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.984202    5689 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 05:03:41.997418 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.984204    5689 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 05:03:41.997902 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.984231    5689 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 05:03:41.997902 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985303    5689 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 05:03:41.997902 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985320    5689 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 05:03:41.997902 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985326    5689 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 05:03:41.997902 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985331    5689 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 05:03:41.997902 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985335    5689 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 05:03:41.997902 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985339    5689 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 05:03:41.997902 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985344    5689 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 05:03:41.997902 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985348    5689 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 05:03:41.997902 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985359    5689 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 05:03:41.997902 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985363    5689 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 05:03:41.997902 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985367    5689 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 05:03:41.997902 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985372    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 05:03:41.997902 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985376    5689 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 05:03:41.997902 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985380    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 05:03:41.997902 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985385    5689 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 05:03:41.998283 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985389    5689 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 05:03:41.998283 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985393    5689 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 05:03:41.998283 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985398    5689 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 05:03:41.998283 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985402    5689 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 05:03:41.998283 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985406    5689 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 05:03:41.998283 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985414    5689 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 05:03:41.998283 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985467    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 05:03:41.998283 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985473    5689 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 05:03:41.998283 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985478    5689 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 05:03:41.998283 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985482    5689 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 05:03:41.998283 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985486    5689 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 05:03:41.998283 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985490    5689 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 05:03:41.998283 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985530    5689 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 05:03:41.998283 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985555    5689 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 05:03:41.998283 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985559    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 05:03:41.998283 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985563    5689 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 05:03:41.998283 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985566    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 05:03:41.998283 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985569    5689 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 05:03:41.998283 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985572    5689 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 05:03:41.998845 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985575    5689 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 05:03:41.998845 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985578    5689 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 05:03:41.998845 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985580    5689 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 05:03:41.998845 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985583    5689 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 05:03:41.998845 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985587    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 05:03:41.998845 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985589    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 05:03:41.998845 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985593    5689 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 05:03:41.998845 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985599    5689 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 05:03:41.998845 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985601    5689 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 05:03:41.998845 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985604    5689 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 05:03:41.998845 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985607    5689 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 05:03:41.998845 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985610    5689 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 05:03:41.998845 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985620    5689 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 05:03:41.998845 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985718    5689 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 05:03:41.998845 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985722    5689 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 05:03:41.998845 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985725    5689 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 05:03:41.998845 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985727    5689 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 05:03:41.998845 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985730    5689 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 05:03:41.998845 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985735    5689 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 05:03:41.999308 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985742    5689 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 05:03:41.999308 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985745    5689 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 05:03:41.999308 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985748    5689 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 05:03:41.999308 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985751    5689 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 05:03:41.999308 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985754    5689 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 05:03:41.999308 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985757    5689 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 05:03:41.999308 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985760    5689 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 05:03:41.999308 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985762    5689 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 05:03:41.999308 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985765    5689 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 05:03:41.999308 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985768    5689 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 05:03:41.999308 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985770    5689 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 05:03:41.999308 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985774    5689 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 05:03:41.999308 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985778    5689 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 05:03:41.999308 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985781    5689 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 05:03:41.999308 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985784    5689 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 05:03:41.999308 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985786    5689 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 05:03:41.999308 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985789    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 05:03:41.999308 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985792    5689 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 05:03:41.999308 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985794    5689 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 05:03:41.999910 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985797    5689 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 05:03:41.999910 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985800    5689 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 05:03:41.999910 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985804    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 05:03:41.999910 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985807    5689 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 05:03:41.999910 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985809    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 05:03:41.999910 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985812    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 05:03:41.999910 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985815    5689 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 05:03:41.999910 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985818    5689 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 05:03:41.999910 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985820    5689 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 05:03:41.999910 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985823    5689 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 05:03:41.999910 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985826    5689 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 05:03:41.999910 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985828    5689 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 05:03:41.999910 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985831    5689 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 05:03:41.999910 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985834    5689 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 05:03:41.999910 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985837    5689 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 05:03:41.999910 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985839    5689 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 05:03:41.999910 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985842    5689 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 05:03:41.999910 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985844    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 05:03:41.999910 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985847    5689 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 05:03:41.999910 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985850    5689 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 05:03:42.000417 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985852    5689 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 05:03:42.000417 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985855    5689 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 05:03:42.000417 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985858    5689 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 05:03:42.000417 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985861    5689 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 05:03:42.000417 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985864    5689 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 05:03:42.000417 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985867    5689 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 05:03:42.000417 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985872    5689 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 05:03:42.000417 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985874    5689 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 05:03:42.000417 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985877    5689 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 05:03:42.000417 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985880    5689 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 05:03:42.000417 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985883    5689 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 05:03:42.000417 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985885    5689 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 05:03:42.000417 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985888    5689 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 05:03:42.000417 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985891    5689 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 05:03:42.000417 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.985927    5689 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 05:03:42.000417 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985974    5689 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 05:03:42.000830 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985981    5689 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 05:03:42.000830 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985984    5689 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 05:03:42.000830 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985987    5689 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 05:03:42.000830 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985989    5689 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 05:03:42.000830 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985992    5689 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 05:03:42.000830 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985994    5689 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 05:03:42.000830 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985997    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 05:03:42.000830 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.985999    5689 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 05:03:42.000830 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986003    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 05:03:42.000830 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986006    5689 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 05:03:42.000830 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986008    5689 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 05:03:42.000830 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986011    5689 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 05:03:42.000830 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986013    5689 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 05:03:42.000830 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986016    5689 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 05:03:42.000830 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986018    5689 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 05:03:42.000830 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986021    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 05:03:42.000830 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986023    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 05:03:42.000830 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986026    5689 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 05:03:42.000830 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986028    5689 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 05:03:42.001384 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986031    5689 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 05:03:42.001384 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986034    5689 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 05:03:42.001384 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986036    5689 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 05:03:42.001384 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986039    5689 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 05:03:42.001384 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986041    5689 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 05:03:42.001384 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986044    5689 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 05:03:42.001384 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986047    5689 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 05:03:42.001384 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986049    5689 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 05:03:42.001384 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986052    5689 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 05:03:42.001384 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986054    5689 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 05:03:42.001384 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986058    5689 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 05:03:42.001384 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986061    5689 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 05:03:42.001384 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986065    5689 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 05:03:42.001384 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986067    5689 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 05:03:42.001384 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986070    5689 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 05:03:42.001384 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986072    5689 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 05:03:42.001384 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986075    5689 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 05:03:42.001384 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986077    5689 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 05:03:42.001384 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986080    5689 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 05:03:42.002199 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986082    5689 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 05:03:42.002199 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986085    5689 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 05:03:42.002199 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986087    5689 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 05:03:42.002199 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986092    5689 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 05:03:42.002199 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986096    5689 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 05:03:42.002199 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986099    5689 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 05:03:42.002199 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986103    5689 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 05:03:42.002199 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986105    5689 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 05:03:42.002199 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986108    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 05:03:42.002199 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986111    5689 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 05:03:42.002199 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986115    5689 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 05:03:42.002199 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986118    5689 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 05:03:42.002199 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986121    5689 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 05:03:42.002199 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986123    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 05:03:42.002199 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986126    5689 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 05:03:42.002199 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986128    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 05:03:42.002199 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986131    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 05:03:42.002199 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986133    5689 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 05:03:42.002199 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986136    5689 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 05:03:42.002914 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986138    5689 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 05:03:42.002914 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986141    5689 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 05:03:42.002914 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986143    5689 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 05:03:42.002914 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986145    5689 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 05:03:42.002914 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986148    5689 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 05:03:42.002914 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986150    5689 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 05:03:42.002914 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986153    5689 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 05:03:42.002914 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986155    5689 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 05:03:42.002914 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986158    5689 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 05:03:42.002914 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986160    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 05:03:42.002914 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986162    5689 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 05:03:42.002914 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986165    5689 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 05:03:42.002914 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986167    5689 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 05:03:42.002914 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986171    5689 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 05:03:42.002914 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986174    5689 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 05:03:42.002914 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986176    5689 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 05:03:42.002914 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986179    5689 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 05:03:42.002914 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986182    5689 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 05:03:42.002914 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986184    5689 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 05:03:42.002914 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986187    5689 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 05:03:42.003390 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986189    5689 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 05:03:42.003390 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986192    5689 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 05:03:42.003390 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986194    5689 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 05:03:42.003390 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986197    5689 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 05:03:42.003390 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986200    5689 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 05:03:42.003390 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986203    5689 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 05:03:42.003390 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986205    5689 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 05:03:42.003390 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986208    5689 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 05:03:42.003390 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986210    5689 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 05:03:42.003390 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986212    5689 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 05:03:42.003390 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986215    5689 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 05:03:42.003390 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986217    5689 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 05:03:42.003390 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986220    5689 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 05:03:42.003390 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986222    5689 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 05:03:42.003390 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986224    5689 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 05:03:42.003390 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986227    5689 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 05:03:42.003390 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986229    5689 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 05:03:42.003390 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986232    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 05:03:42.003390 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986234    5689 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 05:03:42.003390 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986236    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 05:03:42.003929 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986241    5689 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 05:03:42.003929 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986244    5689 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 05:03:42.003929 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986246    5689 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 05:03:42.003929 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986249    5689 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 05:03:42.003929 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986251    5689 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 05:03:42.003929 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986254    5689 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 05:03:42.003929 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986256    5689 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 05:03:42.003929 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:41.986260    5689 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 05:03:42.003929 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.986319    5689 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 01 05:03:42.003929 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.987279    5689 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 01 05:03:42.003929 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.987336    5689 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 01 05:03:42.003929 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.987532    5689 server.go:1014] "Starting client certificate rotation"
Nov 01 05:03:42.003929 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.987664    5689 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 01 05:03:42.003929 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.987709    5689 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 04:57:29 +0000 UTC" deadline="2028-03-27 00:05:56.369848974 +0000 UTC"
Nov 01 05:03:42.003929 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.987737    5689 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12283h2m14.38211337s"
Nov 01 05:03:42.003929 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.987984    5689 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 05:03:42.004331 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.989076    5689 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 05:03:42.004331 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.990308    5689 log.go:25] "Validated CRI v1 runtime API"
Nov 01 05:03:42.004331 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.992230    5689 log.go:25] "Validated CRI v1 image API"
Nov 01 05:03:42.004331 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.993014    5689 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 01 05:03:42.004331 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.995290    5689 fs.go:135] Filesystem UUIDs: map[31536baf-b501-4d7f-869c-69829feada7c:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 8c206422-f772-4760-9f25-c27d9444a6f6:/dev/nvme0n1p4]
Nov 01 05:03:42.004509 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:41.995310    5689 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0711f3d4e1d586814ad4969dd3f79f2539b26f54e5db605f7723168fd44db9d8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0711f3d4e1d586814ad4969dd3f79f2539b26f54e5db605f7723168fd44db9d8/userdata/shm major:0 minor:224 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0a973929be5cebc219b720e4cbc988c2101c4d3885f1252d602d91ece8d10c9b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0a973929be5cebc219b720e4cbc988c2101c4d3885f1252d602d91ece8d10c9b/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1d7634964deac3f00ec41b31563aff4f6799d2d66d08ad1fcfed920814d877a6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1d7634964deac3f00ec41b31563aff4f6799d2d66d08ad1fcfed920814d877a6/userdata/shm major:0 minor:413 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/35f50f1e7eed82a9c567aa5d24724ef38198719bbf0c8a47d237d2da9df7aaa5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/35f50f1e7eed82a9c567aa5d24724ef38198719bbf0c8a47d237d2da9df7aaa5/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4255f96b8c68d9e9d843b9faaca419295cf827087a981964731ed862bdadda7c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4255f96b8c68d9e9d843b9faaca419295cf827087a981964731ed862bdadda7c/userdata/shm major:0 minor:300 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/46168ed50cbd4d5ada910d8a21a69f2d8d81f28884def6ccc1f353dd392a4090/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/46168ed50cbd4d5ada910d8a21a69f2d8d81f28884def6ccc1f353dd392a4090/userdata/shm major:0 minor:259 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4c6b70bec32c1ece062d94681b27d82605927e2b81b18ab88589b471bb233102/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4c6b70bec32c1ece062d94681b27d82605927e2b81b18ab88589b471bb233102/userdata/shm major:0 minor:264 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4eff5ab2517b779c97c79777f957957fe446c9b09aba3853080f6cc1edf21822/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4eff5ab2517b779c97c79777f957957fe446c9b09aba3853080f6cc1edf21822/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/53da545c04e8752f739145afdc125ac6d8539e8e74e0931230115f6db75d4ff7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/53da545c04e8752f739145afdc125ac6d8539e8e74e0931230115f6db75d4ff7/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5bc3adeb1375285df89458f45d2042ba303bd03e91b726e1d832adcc059030f9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5bc3adeb1375285df89458f45d2042ba303bd03e91b726e1d832adcc059030f9/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5db42fabf816db16908b2a5afc5dbc7358a7c1d9679a9bbe1be0cdc2008a9469/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5db42fabf816db16908b2a5afc5dbc7358a7c1d9679a9bbe1be0cdc2008a9469/userdata/shm major:0 minor:304 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6f360fd45dad65dd6e7d7d898de884c382f2d686f42036ee1b5d1f749bb636d9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6f360fd45dad65dd6e7d7d898de884c382f2d686f42036ee1b5d1f749bb636d9/userdata/shm major:0 minor:330 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/77d30d727f12a09cd823c280b3f077ffd3a480b47977d9f32f3b0ab9be8b4cb5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/77d30d727f12a09cd823c280b3f077ffd3a480b47977d9f32f3b0ab9be8b4cb5/userdata/shm major:0 minor:218 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/78cfa2ceed0dfad5311f2c8230151b15c354cce14a4b7b45aa09dac43e4efa53/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/78cfa2ceed0dfad5311f2c8230151b15c354cce14a4b7b45aa09dac43e4efa53/userdata/shm major:0 minor:302 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7e06150d8e2d064f862ad246e3da4c8cb964e2306c18fa4b2280dd90c1587af7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7e06150d8e2d064f862ad246e3da4c8cb964e2306c18fa4b2280dd90c1587af7/userdata/shm major:0 minor:63 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/941920de309566d6056dbe93be54c9881a2d6a53c9d51e866cf373cb386906f9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/941920de309566d6056dbe93be54c9881a2d6a53c9d51e866cf373cb386906f9/userdata/shm major:0 minor:349 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/94b090bb2f81614395a6c92d20a127033288b6317a3746960c12ca9c06f937c8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/94b090bb2f81614395a6c92d20a127033288b6317a3746960c12ca9c06f937c8/userdata/shm major:0 minor:341 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a4e2086e72083ac7db69643e5261a496346ab4a08e69339adc535572a63c94f4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a4e2086e72083ac7db69643e5261a496346ab4a08e69339adc535572a63c94f4/userdata/shm major:0 minor:375 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a6cb025b09c74b2375547c39e2bcf11018d0fe067f47fa5bc687270d83d3c4d5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a6cb025b09c74b2375547c39e2bcf11018d0fe067f47fa5bc687270d83d3c4d5/userdata/shm major:0 minor:334 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d1f3f2d5ced62268ee33102b7bc84b719b90c807b6ef49c26c5eec98182f2d13/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d1f3f2d5ced62268ee33102b7bc84b719b90c807b6ef49c26c5eec98182f2d13/userdata/shm major:0 minor:211 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d69ee7d9a731d0de5d92c0c9c962a0915b8fe7820c45d13c2b059af4a12e65ea/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d69ee7d9a731d0de5d92c0c9c962a0915b8fe7820c45d13c2b059af4a12e65ea/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e4b4d6c6a74ba41ccfa800406dd6347bda703d9e8cde5971cf0cf9f4b1d1e118/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e4b4d6c6a74ba41ccfa800406dd6347bda703d9e8cde5971cf0cf9f4b1d1e118/userdata/shm major:0 minor:354 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e4e554abfa6a31cdecc3e57264ae7ce20607c5d8b75e65f9b6d2bdd3c3d435e9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e4e554abfa6a31cdecc3e57264ae7ce20607c5d8b75e65f9b6d2bdd3c3d435e9/userdata/shm major:0 minor:77 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e7b086d9337668f71e0f1d261a7126a33c053fb5fa12c69ccd7eb3548c960812/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e7b086d9337668f71e0f1d261a7126a33c053fb5fa12c69ccd7eb3548c960812/userdata/shm major:0 minor:62 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e96638928e69534905932599d19846f2632b24018a9b807eb5d9c55f0b84bc3a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e96638928e69534905932599d19846f2632b24018a9b807eb5d9c55f0b84bc3a/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/efed176f934c493edca2ddc32acd11fa4382cd5265b4f1516ff814fbb11a2114/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/efed176f934c493edca2ddc32acd11fa4382cd5265b4f1516ff814fbb11a2114/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fa82b731dbe7fc0fedf4b1e1fc8c3c6dca92546a0a2b97db3d83e7f320bc0bd3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fa82b731dbe7fc0fedf4b1e1fc8c3c6dca92546a0a2b97db3d83e7f320bc0bd3/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0b0122f6-d51a-4338-a631-056ec9311b69/volumes/kubernetes.io~projected/kube-api-access-9njs4:{mountpoint:/var/lib/kubelet/pods/0b0122f6-d51a-4338-a631-056ec9311b69/volumes/kubernetes.io~projected/kube-api-access-9njs4 major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0b0122f6-d51a-4338-a631-056ec9311b69/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/0b0122f6-d51a-4338-a631-056ec9311b69/volumes/kubernetes.io~secret/metrics-tls major:0 minor:254 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6/volumes/kubernetes.io~projected/kube-api-access-55zt7:{mountpoint:/var/lib/kubelet/pods/0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6/volumes/kubernetes.io~projected/kube-api-access-55zt7 major:0 minor:412 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:406 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:411 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:410 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0e24f687-655b-4edb-8901-35412d3fc79b/volumes/kubernetes.io~projected/kube-api-access-tgfc6:{mountpoint:/var/lib/kubelet/pods/0e24f687-655b-4edb-8901-35412d3fc79b/volumes/kubernetes.io~projected/kube-api-access-tgfc6 major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1bd1166d-68e9-4035-b920-e42f5bf05d11/volumes/kubernetes.io~projected/kube-api-access-7npk7:{mountpoint:/var/lib/kubelet/pods/1bd1166d-68e9-4035-b920-e42f5bf05d11/volumes/kubernetes.io~projected/kube-api-access-7npk7 major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1bd1166d-68e9-4035-b920-e42f5bf05d11/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/1bd1166d-68e9-4035-b920-e42f5bf05d11/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:339 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1bd1166d-68e9-4035-b920-e42f5bf05d11/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/1bd1166d-68e9-4035-b920-e42f5bf05d11/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:338 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/35391db2-ab6e-4272-8580-9014e24f72c4/volumes/kubernetes.io~secret/tls-certificates:{mountpoint:/var/lib/kubelet/pods/35391db2-ab6e-4272-8580-9014e24f72c4/volumes/kubernetes.io~secret/tls-certificates major:0 minor:292 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4321e0ae-e339-40f2-875a-941b1183e766/volumes/kubernetes.io~projected/kube-api-access-2wlnj:{mountpoint:/var/lib/kubelet/pods/4321e0ae-e339-40f2-875a-941b1183e766/volumes/kubernetes.io~projected/kube-api-access-2wlnj major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4321e0ae-e339-40f2-875a-941b1183e766/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/4321e0ae-e339-40f2-875a-941b1183e766/volumes/kubernetes.io~secret/metrics-certs major:0 minor:344 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/46078cac-ae78-4ac7-bda3-b95913a478f8/volumes/kubernetes.io~projected/kube-api-access-djmg6:{mountpoint:/var/lib/kubelet/pods/46078cac-ae78-4ac7-bda3-b95913a478f8/volumes/kubernetes.io~projected/kube-api-access-djmg6 major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/46078cac-ae78-4ac7-bda3-b95913a478f8/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/46078cac-ae78-4ac7-bda3-b95913a478f8/volumes/kubernetes.io~secret/signing-key major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6149f269-49c2-4572-8f6c-fe01e9f0dcbf/volumes/kubernetes.io~projected/kube-api-access-ntxmg:{mountpoint:/var/lib/kubelet/pods/6149f269-49c2-4572-8f6c-fe01e9f0dcbf/volumes/kubernetes.io~projected/kube-api-access-ntxmg major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6f4aae57-6429-4f9e-aaa5-c700bc26b4f5/volumes/kubernetes.io~projected/kube-api-access-248gt:{mountpoint:/var/lib/kubelet/pods/6f4aae57-6429-4f9e-aaa5-c700bc26b4f5/volumes/kubernetes.io~projected/kube-api-access-248gt major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7cd84f11-44c9-4ec6-83c2-7db17ab9991c/volumes/kubernetes.io~projected/kube-api-access-8mg57:{mountpoint:/var/lib/kubelet/pods/7cd84f11-44c9-4ec6-83c2-7db17ab9991c/volumes/kubernetes.io~projected/kube-api-access-8mg57 major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7e9cccdd-6c18-4a60-837f-4907411a1c44/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/7e9cccdd-6c18-4a60-837f-4907411a1c44/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7e9cccdd-6c18-4a60-837f-4907411a1c44/volumes/kubernetes.io~projected/kube-api-access-tmrx6:{mountpoint:/var/lib/kubelet/pods/7e9cccdd-6c18-4a60-837f-4907411a1c44/volumes/kubernetes.io~projected/kube-api-access-tmrx6 major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7e9cccdd-6c18-4a60-837f-4907411a1c44/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/7e9cccdd-6c18-4a60-837f-4907411a1c44/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/805c87a3-63e9-4c13-9c90-c05f73282edc/volumes/kubernetes.io~projected/kube-api-access-hvgxl:{mountpoint:/var/lib/kubelet/pods/805c87a3-63e9-4c13-9c90-c05f73282edc/volumes/kubernetes.io~projected/kube-api-access-hvgxl major:0 minor:329 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/805c87a3-63e9-4c13-9c90-c05f73282edc/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/805c87a3-63e9-4c13-9c90-c05f73282edc/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:326 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/805c87a3-63e9-4c13-9c90-c05f73282edc/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/805c87a3-63e9-4c13-9c90-c05f73282edc/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:327 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/80c10aac-5175-46d6-a18b-109169a2461b/volumes/kubernetes.io~projected/kube-api-access-c7hhr:{mountpoint:/var/lib/kubelet/pods/80c10aac-5175-46d6-a18b-109169a2461b/volumes/kubernetes.io~projected/kube-api-access-c7hhr major:0 minor:374 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/80c10aac-5175-46d6-a18b-109169a2461b/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/80c10aac-5175-46d6-a18b-109169a2461b/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:369 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/80c10aac-5175-46d6-a18b-109169a2461b/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/80c10aac-5175-46d6-a18b-109169a2461b/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:371 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/80c10aac-5175-46d6-a18b-109169a2461b/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/80c10aac-5175-46d6-a18b-109169a2461b/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:367 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/80c10aac-5175-46d6-a18b-109169a2461b/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/80c10aac-5175-46d6-a18b-109169a2461b/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:368 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/80c10aac-5175-46d6-a18b-109169a2461b/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/80c10aac-5175-46d6-a18b-109169a2461b/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:370 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/80c10aac-5175-46d6-a18b-109169a2461b/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/80c10aac-5175-46d6-a18b-109169a2461b/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:372 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8fba6aca-0a63-4924-89b2-87575d844ce6/volumes/kubernetes.io~projected/kube-api-access-s6nj8:{mountpoint:/var/lib/kubelet/pods/8fba6aca-0a63-4924-89b2-87575d844ce6/volumes/kubernetes.io~projected/kube-api-access-s6nj8 major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8fba6aca-0a63-4924-89b2-87575d844ce6/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/8fba6aca-0a63-4924-89b2-87575d844ce6/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:263 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/972b129f-0166-4705-a71c-09ee0e7a91b9/volumes/kubernetes.io~projected/kube-api-access-lv27z:{mountpoint:/var/lib/kubelet/pods/972b129f-0166-4705-a71c-09ee0e7a91b9/volumes/kubernetes.io~projected/kube-api-access-lv27z major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9752ded2-725f-4035-b374-602e4b4eb576/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/9752ded2-725f-4035-b374-602e4b4eb576/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9752ded2-725f-4035-b374-602e4b4eb576/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/9752ded2-725f-4035-b374-602e4b4eb576/volumes/kubernetes.io~empty-dir/tmp major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9752ded2-725f-4035-b374-602e4b4eb576/volumes/kubernetes.io~projected/kube-api-access-8rrgq:{mountpoint:/var/lib/kubelet/pods/9752ded2-725f-4035-b374-602e4b4eb576/volumes/kubernetes.io~projected/kube-api-access-8rrgq major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14/volumes/kubernetes.io~projected/kube-api-access-r5zxr:{mountpoint:/var/lib/kubelet/pods/9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14/volumes/kubernetes.io~projected/kube-api-access-r5zxr major:0 minor:328 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:320 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:325 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a877f895-a4cf-404d-87e4-440d4b73a870/volumes/kubernetes.io~projected/kube-api-access-xc8bg:{mountpoint:/var/lib/kubelet/pods/a877f895-a4cf-404d-87e4-440d4b73a870/volumes/kubernetes.io~projected/kube-api-access-xc8bg major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a8db8643-51bf-4bcf-b26d-bd937a67b5ed/volumes/kubernetes.io~projected/kube-api-access-4zn2l:{mountpoint:/var/lib/kubelet/pods/a8db8643-51bf-4bcf-b26d-bd937a67b5ed/volumes/kubernetes.io~projected/kube-api-access-4zn2l major:0 minor:299 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a9bb6df5-0993-4640-81cc-52426b72fb3e/volumes/kubernetes.io~projected/kube-api-access-n95qs:{mountpoint:/var/lib/kubelet/pods/a9bb6df5-0993-4640-81cc-52426b72fb3e/volumes/kubernetes.io~projected/kube-api-access-n95qs major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/af0da37a-a117-49a6-a06e-6cab0e56a40c/volumes/kubernetes.io~projected/kube-api-access-hd22d:{mountpoint:/var/lib/kubelet/pods/af0da37a-a117-49a6-a06e-6cab0e56a40c/volumes/kubernetes.io~projected/kube-api-access-hd22d major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/af0da37a-a117-49a6-a06e-6cab0e56a40c/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/af0da37a-a117-49a6-a06e-6cab0e56a40c/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bcc64278-da5e-4525-90bf-abddde870d70/volumes/kubernetes.io~projected/kube-api-access-ffjr4:{mountpoint:/var/lib/kubelet/pods/bcc64278-da5e-4525-90bf-abddde870d70/volumes/kubernetes.io~projected/kube-api-access-ffjr4 major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ca43c15d-cefb-4e78-93aa-a98573da4feb/volumes/kubernetes.io~projected/kube-api-access-wgwkh:{mountpoint:/var/lib/kubelet/pods/ca43c15d-cefb-4e78-93aa-a98573da4feb/volumes/kubernetes.io~projected/kube-api-access-wgwkh major:0 minor:298 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ca43c15d-cefb-4e78-93aa-a98573da4feb/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/ca43c15d-cefb-4e78-93aa-a98573da4feb/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:296 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ca43c15d-cefb-4e78-93aa-a98573da4feb/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/ca43c15d-cefb-4e78-93aa-a98573da4feb/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:297 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cca47ebc-569a-4558-af41-5e3755605b9d/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/cca47ebc-569a-4558-af41-5e3755605b9d/volumes/kubernetes.io~secret/agent-certs major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d454e879-03a2-4fb9-8223-79a1aa9d85d2/volumes/kubernetes.io~projected/kube-api-access-hdlnm:{mountpoint:/var/lib/kubelet/pods/d454e879-03a2-4fb9-8223-79a1aa9d85d2/volumes/kubernetes.io~projected/kube-api-access-hdlnm major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e856df5a-5c5e-4421-ae7d-80a760bc72e2/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/e856df5a-5c5e-4421-ae7d-80a760bc72e2/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:343 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/37ca77d9f5e44b9a3d33c4ee54982f42b402a4b7be12cfb11ca9876753c3e204/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/f796d2b199c6c0af8deb099a1133cf2794ae20fb21ae7a01ddfeb62467c866cf/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/e1605d24806a6e367b95e4e434c78580cd575d30cf2fdb4dd6eb4e2941ce707b/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/dac7c3cbdb9516696d67957949902e20b86ce66b1a7005f061f09b5ea7f66a57/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/b43bcec35fff21d65585042ed503f84ba5b7209e1ef34300f7bcb6c22a2c25cf/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/6541bf6c76c26dfa44424af39c30cbb046a94c19ce8fe1cd4dfd06e31a4e7a33/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/839031eba93b354434fd88a91d4c6ea2dbe559dc12631c09c447c1b5ffd02031/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/6f88d3e842ba81df2a1c74426bf2cca010b3a41d886571b5fa3ef135414b6a7d/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/fffc6298c319cf5c04b1cc3ad109787b7c435b5e00d80216ece2bd4adad7bc07/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/526fe9b033e6cce386cdba92e849cdd8b0ae95ec194264d4ab382557b2b2bae1/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-135:{mountpoint:/var/lib/containers/storage/overlay/d71f17f4c2205c48d4302011102a08be90c97be06023abddf5f09d76a4ff2b52/merged major:0 minor:135 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/23972949cdfddd5eaaa1580bc489220c4d726b7257ac2123ca21608f7f88ca7c/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/8ed144a8f67874dc4c18178806b2c31dc4563461422df449a2b771556ff9cd78/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/7e33dd2f0736003159df681167300f6b2a05211bd56f91d8b809611b82b57718/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/5d93222bead04ec2b241a0178271850b2e154ca7470adccca5cbea8b67b25d59/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/0a8d87b262ed8de8c3a09ce7d457efdb3c8bd145233d54c4e65ddeb21185a3e3/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-174:{mountpoint:/var/lib/containers/storage/overlay/4bb5881139d32ffd7682529fe8fbbd1d3edaf8e21640aff648d30a0f9a19f5ff/merged major:0 minor:174 fsType:overlay blockSize:0} overlay_0-179:{mountpoint:/var/lib/containers/storage/overlay/d30823c2af96ebc660103113bcc38dd44dfef4ddf8ec99df128c1385a46147ca/merged major:0 minor:179 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/5a053f268dfc241cf044bb49c2d770ead69e4d00c679311221b5ef1fc6b41283/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/b28f8ef3f4988491e08f3ccb35a3fffe775baf9f7e59628b2dbb22c60d5b1dad/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/627c2ae29fda845c950aafddf09b3d250343f65ed051ad7ff9ac919a1f53422a/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/aa96fcf3afcc423a4e91892bac98392539b5c86f0c719615a71e8d7e43603cd8/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/9affd74e5fe5197d5f20aed89b1df1c0079dc5964081a68a5a0a8d2b81428c51/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/c0c996b3383c48e3214e6b90ea1d401b8de968f1e8e0cc58fc5df681f6799321/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-220:{mountpoint:/var/lib/containers/storage/overlay/bdcea272e50399dea2b78502496a15035c65e5e75fca20b5b56c5f886c97e4a0/merged major:0 minor:220 fsType:overlay blockSize:0} overlay_0-226:{mountpoint:/var/lib/containers/storage/overlay/0c0491ea4e30bc7f8c90dc23c48472ba9023be3cf314758e3728cc3627cb7914/merged major:0 minor:226 fsType:overlay blockSize:0} overlay_0-228:{mountpoint:/var/lib/containers/storage/overlay/56b91b45e815223c9a6cac74a8f7b93a75593e72aa19aabd32bd4964fdba61f9/merged major:0 minor:228 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/60a3962592cfe0ca71906072eeba1b29d6d119e70542c9d3f417003f7dadf5af/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/304277ab573bbc3714dc3d9e43f547d9296d5f9397bcb96da9be76187998639c/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-242:{mountpoint:/var/lib/containers/storage/overlay/c4bf9fba5d1c342ce0883502a387585edaeea63fb1e9c0d9896f3cb1abd999b7/merged major:0 minor:242 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/0f3daa450906c32051de52976b830a40e8b8a4edd79b8a549549179799fec1dd/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-261:{mountpoint:/var/lib/containers/storage/overlay/5497e9cf89dc9ebe04f126298da38714ecb649d1e7f02394e6d874f7ff470d76/merged major:0 minor:261 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/ef81752ad35195ee976685e797add3e4a75a9cbeda20bfc78e470f65316b88fd/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/fff79a933a2e5cb6c6075cf5f57ed2523a8f5c5f8034587fe2ec984ce1e58c34/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/b1f23d54192f216c97c5c054e02dd1dda3c998d5446e0fc976ddeae8b799264d/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/636ebcbb0fc106ff23877546417f015fb40d7a443a465082ff5dff463c8dbe92/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-281:{mountpoint:/var/lib/containers/storage/overlay/d935f131a00042aa2906a6b47736b8c35807bdb71af57fe34d808807148c65e5/merged major:0 minor:281 fsType:overlay blockSize:0} overlay_0-287:{mountpoint:/var/lib/containers/storage/overlay/be9566066cffb23d7872adfdfd4e477f9e1302c30562ee0520573d5a79a4fe8f/merged major:0 minor:287 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/418185dafe1bf4dcd5d6e3c2b98c7ad12c812cbd8cfd16929714b1a20d944d1b/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/0a74611428f166a013b6cc11d9d850b637d0143d4cefd925bf0600b5851f6ff6/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/aafea5f2f8aad10f928296e15b3338250b988766304183c10c05ac5a971b7622/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-312:{mountpoint:/var/lib/containers/storage/overlay/09a19d335bb8cb3cd3cabbcee12c2c4b03154dab72ed42424e73c22df938c401/merged major:0 minor:312 fsType:overlay blockSize:0} overlay_0-314:{mountpoint:/var/lib/containers/storage/overlay/0b116c1cb12a3b74d187ffce9da0731603ac90e8f76184b4a753fd196d579112/merged major:0 minor:314 fsType:overlay blockSize:0} overlay_0-332:{mountpoint:/var/lib/containers/storage/overlay/0728c95d7baf6d81d5cd3902966ff4a0b8fa6b30113af58eb3234feed57d78c7/merged major:0 minor:332 fsType:overlay blockSize:0} overlay_0-336:{mountpoint:/var/lib/containers/storage/overlay/6f825c62ed5e455db44e76b61934909cd0e0df369abb253ff01bc4e48e3278f6/merged major:0 minor:336 fsType:overlay blockSize:0} overlay_0-345:{mountpoint:/var/lib/containers/storage/overlay/f0727a5d60344e6dd00624cf5c439b611d2e2953226b6db4665ccc3e7658bc10/merged major:0 minor:345 fsType:overlay blockSize:0} overlay_0-347:{mountpoint:/var/lib/containers/storage/overlay/ea0b0e4d596b46f7fa31dc391f1e563d0d5a043095086118ce9c5dd0b66a997c/merged major:0 minor:347 fsType:overlay blockSize:0} overlay_0-357:{mountpoint:/var/lib/containers/storage/overlay/7a8e6b949a7dcb921b03f82e96cf6196331c60e52256c7fbd7943e8290599d28/merged major:0 minor:357 fsType:overlay blockSize:0} overlay_0-359:{mountpoint:/var/lib/containers/storage/overlay/f3b1c86dfb02b7ae9b2f49ad1552564567e2c251841a13e54ce672f8222f63b2/merged major:0 minor:359 fsType:overlay blockSize:0} overlay_0-361:{mountpoint:/var/lib/containers/storage/overlay/55ba2f60246ca623f34be05bd0ef1e2431405650ebfc2436509b8896321334bb/merged major:0 minor:361 fsType:overlay blockSize:0} overlay_0-363:{mountpoint:/var/lib/containers/storage/overlay/bb9fad70fdf9f3905ae278ed92c6b167844007300125ebb5c46a08e4adf29ded/merged major:0 minor:363 fsType:overlay blockSize:0} overlay_0-380:{mountpoint:/var/lib/containers/storage/overlay/eb7f242d5acdffa0be84df41a6db84002a174efa5b911aad7ae9d82a7a07c018/merged major:0 minor:380 fsType:overlay blockSize:0} overlay_0-382:{mountpoint:/var/lib/containers/storage/overlay/18c8c3d22956560798abe2372f0a8a7d506f4ee11bfeec553576a6f1e6952b04/merged major:0 minor:382 fsType:overlay blockSize:0} overlay_0-388:{mountpoint:/var/lib/containers/storage/overlay/1a4a8f412f1924aa15170772859a67450293eb74bc0184548e136c4e6e9f2603/merged major:0 minor:388 fsType:overlay blockSize:0} overlay_0-393:{mountpoint:/var/lib/containers/storage/overlay/2d38c9205b491a42b7a0cc65d2cf0bb8d24085510f93e9074bb73fa4f5ec2086/merged major:0 minor:393 fsType:overlay blockSize:0} overlay_0-398:{mountpoint:/var/lib/containers/storage/overlay/cc4edec2f77af11bd8d0c3ce6516cf61de7a89c2854954179c97ca14b7039f63/merged major:0 minor:398 fsType:overlay blockSize:0} overlay_0-400:{mountpoint:/var/lib/containers/storage/overlay/868e64f65beb6a32fe41836ff30c6d47c9b6519ea2d878992cf5770ebf0d8f02/merged major:0 minor:400 fsType:overlay blockSize:0} overlay_0-415:{mountpoint:/var/lib/containers/storage/overlay/1f91105497337dbc11c667a7a105378137da07f201ffcc8e85cc91bb1f85671f/merged major:0 minor:415 fsType:overlay blockSize:0} overlay_0-417:{mountpoint:/var/lib/containers/storage/overlay/284ce29d1e190cdab4c03b05b9a3dd81d21bb69115f6566527a595b529fb1e41/merged major:0 minor:417 fsType:overlay blockSize:0} overlay_0-419:{mountpoint:/var/lib/containers/storage/overlay/cd31c46750286ede6f5fbc447440b951c27a3cc82dc4ec5285782b904f96fb16/merged major:0 minor:419 fsType:overlay blockSize:0} overlay_0-425:{mountpoint:/var/lib/containers/storage/overlay/11c5b165ff83cedde81f09b598fda3aefc2a8c2c196d551c573f60991a456a17/merged major:0 minor:425 fsType:overlay blockSize:0} overlay_0-430:{mountpoint:/var/lib/containers/storage/overlay/4183d596d94cf7629dc7daaac0be9565354a5813940cba9f3783117f5fccf899/merged major:0 minor:430 fsType:overlay blockSize:0} overlay_0-435:{mountpoint:/var/lib/containers/storage/overlay/0e3ff11e786c5095f9061a56831394934c8c5b555c5a0c079e5a34475b39592f/merged major:0 minor:435 fsType:overlay blockSize:0} overlay_0-441:{mountpoint:/var/lib/containers/storage/overlay/e9e3e927fc28a131043f9dbbe6f198ca19db3220c3f106be18cdb7fdd3f4d659/merged major:0 minor:441 fsType:overlay blockSize:0} overlay_0-446:{mountpoint:/var/lib/containers/storage/overlay/ff195772526749bf0a1807c7ce24b154c601b110ff5e5a38d0d6ea224fa639e8/merged major:0 minor:446 fsType:overlay blockSize:0} overlay_0-451:{mountpoint:/var/lib/containers/storage/overlay/bc88072b833011e348b823d33681441577dd8c5a6240616a8b22687d5bcb7d1f/merged major:0 minor:451 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/7ec58fdc27a7fc03912d5bf6b7356e647d1d58434a0354bee8ea28dbef32eb1f/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/ee6eefb3867944aecdab2fb6706529a8bd52bc9d1a44ac3194f3bd2f2d40fb8a/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/9bd9030bfe0de8f04f9c6b4e834183d3b52220dbd1c4dfe327253871fd1eb89d/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/d4d917e27e341f44369653982e54e7987c195d1ae341fecb043ad499392ea391/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-81:{mountpoint:/var/lib/containers/storage/overlay/69ef54a53ee073514b1f4b97335e206ebe27151710206f419692d0fc53266702/merged major:0 minor:81 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/55e398658508a395c9666c45279165fdf1c79b276f228701e3ac153471520fc4/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/a20553cf2ad4d1ca3c2b90c76381a79544cfffd4c5bdeccb68c2ab4a2c28683b/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/a1906657a184538f8d8bac9e18937447427443812f82c0a0349f4ffec3fc7599/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/c1c8f377b98e9bbb82e95a3254e8c9257ea4e2c579c0fc27d2aa0a5ca8976856/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/f6336af6646b5ce0d8dba26c7368c0dd68ef3a5eb828b75931d68a9b39167895/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 01 05:03:42.004779 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.004244    5689 manager.go:217] Machine: {Timestamp:2026-11-01 05:03:42.00379251 +0000 UTC m=+0.074589222 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100474 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec226b1316f4d753c69b8802ebfe9879 SystemUUID:ec226b13-16f4-d753-c69b-8802ebfe9879 BootID:73ecc392-1f88-480c-adc1-253394410c4e Filesystems:[{Device:overlay_0-228 DeviceMajor:0 DeviceMinor:228 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6f360fd45dad65dd6e7d7d898de884c382f2d686f42036ee1b5d1f749bb636d9/userdata/shm DeviceMajor:0 DeviceMinor:330 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7cd84f11-44c9-4ec6-83c2-7db17ab9991c/volumes/kubernetes.io~projected/kube-api-access-8mg57 DeviceMajor:0 DeviceMinor:207 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-220 DeviceMajor:0 DeviceMinor:220 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1bd1166d-68e9-4035-b920-e42f5bf05d11/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:339 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-135 DeviceMajor:0 DeviceMinor:135 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d1f3f2d5ced62268ee33102b7bc84b719b90c807b6ef49c26c5eec98182f2d13/userdata/shm DeviceMajor:0 DeviceMinor:211 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-451 DeviceMajor:0 DeviceMinor:451 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9752ded2-725f-4035-b374-602e4b4eb576/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:48 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0b0122f6-d51a-4338-a631-056ec9311b69/volumes/kubernetes.io~projected/kube-api-access-9njs4 DeviceMajor:0 DeviceMinor:201 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5db42fabf816db16908b2a5afc5dbc7358a7c1d9679a9bbe1be0cdc2008a9469/userdata/shm DeviceMajor:0 DeviceMinor:304 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-336 DeviceMajor:0 DeviceMinor:336 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e4e554abfa6a31cdecc3e57264ae7ce20607c5d8b75e65f9b6d2bdd3c3d435e9/userdata/shm DeviceMajor:0 DeviceMinor:77 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/53da545c04e8752f739145afdc125ac6d8539e8e74e0931230115f6db75d4ff7/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-380 DeviceMajor:0 DeviceMinor:380 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fa82b731dbe7fc0fedf4b1e1fc8c3c6dca92546a0a2b97db3d83e7f320bc0bd3/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/35f50f1e7eed82a9c567aa5d24724ef38198719bbf0c8a47d237d2da9df7aaa5/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d69ee7d9a731d0de5d92c0c9c962a0915b8fe7820c45d13c2b059af4a12e65ea/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4c6b70bec32c1ece062d94681b27d82605927e2b81b18ab88589b471bb233102/userdata/shm DeviceMajor:0 DeviceMinor:264 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/972b129f-0166-4705-a71c-09ee0e7a91b9/volumes/kubernetes.io~projected/kube-api-access-lv27z DeviceMajor:0 DeviceMinor:58 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7e06150d8e2d064f862ad246e3da4c8cb964e2306c18fa4b2280dd90c1587af7/userdata/shm DeviceMajor:0 DeviceMinor:63 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6f4aae57-6429-4f9e-aaa5-c700bc26b4f5/volumes/kubernetes.io~projected/kube-api-access-248gt DeviceMajor:0 DeviceMinor:208 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/bcc64278-da5e-4525-90bf-abddde870d70/volumes/kubernetes.io~projected/kube-api-access-ffjr4 DeviceMajor:0 DeviceMinor:217 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-357 DeviceMajor:0 DeviceMinor:357 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/ca43c15d-cefb-4e78-93aa-a98573da4feb/volumes/kubernetes.io~projected/kube-api-access-wgwkh DeviceMajor:0 DeviceMinor:298 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14/volumes/kubernetes.io~projected/kube-api-access-r5zxr DeviceMajor:0 DeviceMinor:328 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-382 DeviceMajor:0 DeviceMinor:382 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-345 DeviceMajor:0 DeviceMinor:345 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5bc3adeb1375285df89458f45d2042ba303bd03e91b726e1d832adcc059030f9/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8fba6aca-0a63-4924-89b2-87575d844ce6/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:263 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/805c87a3-63e9-4c13-9c90-c05f73282edc/volumes/kubernetes.io~projected/kube-api-access-hvgxl DeviceMajor:0 DeviceMinor:329 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:411 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/a9bb6df5-0993-4640-81cc-52426b72fb3e/volumes/kubernetes.io~projected/kube-api-access-n95qs DeviceMajor:0 DeviceMinor:54 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0711f3d4e1d586814ad4969dd3f79f2539b26f54e5db605f7723168fd44db9d8/userdata/shm DeviceMajor:0 DeviceMinor:224 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/941920de309566d6056dbe93be54c9881a2d6a53c9d51e866cf373cb386906f9/userdata/shm DeviceMajor:0 DeviceMinor:349 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/80c10aac-5175-46d6-a18b-109169a2461b/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:367 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7e9cccdd-6c18-4a60-837f-4907411a1c44/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-226 DeviceMajor:0 DeviceMinor:226 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:406 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-446 DeviceMajor:0 DeviceMinor:446 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ca43c15d-cefb-4e78-93aa-a98573da4feb/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:297 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/9752ded2-725f-4035-b374-602e4b4eb576/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:47 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-179 DeviceMajor:0 DeviceMinor:179 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8fba6aca-0a63-4924-89b2-87575d844ce6/volumes/kubernetes.io~projected/kube-api-access-s6nj8 DeviceMajor:0 DeviceMinor:206 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-261 DeviceMajor:0 DeviceMinor:261 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e856df5a-5c5e-4421-ae7d-80a760bc72e2/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:343 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/80c10aac-5175-46d6-a18b-109169a2461b/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:370 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/46078cac-ae78-4ac7-bda3-b95913a478f8/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:222 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1bd1166d-68e9-4035-b920-e42f5bf05d11/volumes/kubernetes.io~projected/kube-api-access-7npk7 DeviceMajor:0 DeviceMinor:340 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-347 DeviceMajor:0 DeviceMinor:347 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/80c10aac-5175-46d6-a18b-109169a2461b/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:369 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/80c10aac-5175-46d6-a18b-109169a2461b/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:371 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/cca47ebc-569a-4558-af41-5e3755605b9d/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:50 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a877f895-a4cf-404d-87e4-440d4b73a870/volumes/kubernetes.io~projected/kube-api-access-xc8bg DeviceMajor:0 DeviceMinor:57 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-398 DeviceMajor:0 DeviceMinor:398 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6/volumes/kubernetes.io~projected/kube-api-access-55zt7 DeviceMajor:0 DeviceMinor:412 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-417 DeviceMajor:0 DeviceMinor:417 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7e9cccdd-6c18-4a60-837f-4907411a1c44/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:49 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-81 DeviceMajor:0 DeviceMinor:81 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-287 DeviceMajor:0 DeviceMinor:287 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ca43c15d-cefb-4e78-93aa-a98573da4feb/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:296 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/78cfa2ceed0dfad5311f2c8230151b15c354cce14a4b7b45aa09dac43e4efa53/userdata/shm DeviceMajor:0 DeviceMinor:302 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/80c10aac-5175-46d6-a18b-109169a2461b/volumes/kubernetes.io~projected/kube-api-access-c7hhr DeviceMajor:0 DeviceMinor:374 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/af0da37a-a117-49a6-a06e-6cab0e56a40c/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/77d30d727f12a09cd823c280b3f077ffd3a480b47977d9f32f3b0ab9be8b4cb5/userdata/shm DeviceMajor:0 DeviceMinor:218 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-312 DeviceMajor:0 DeviceMinor:312 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4321e0ae-e339-40f2-875a-941b1183e766/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:344 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/80c10aac-5175-46d6-a18b-109169a2461b/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:368 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-388 DeviceMajor:0 DeviceMinor:388 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:320 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-332 DeviceMajor:0 DeviceMinor:332 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-359 DeviceMajor:0 DeviceMinor:359 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-361 DeviceMajor:0 DeviceMinor:361 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-441 DeviceMajor:0 DeviceMinor:441 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-174 DeviceMajor:0 DeviceMinor:174 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:410 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-435 DeviceMajor:0 DeviceMinor:435 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9752ded2-725f-4035-b374-602e4b4eb576/volumes/kubernetes.io~projected/kube-api-access-8rrgq DeviceMajor:0 DeviceMinor:52 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:325 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/80c10aac-5175-46d6-a18b-109169a2461b/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:372 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-400 DeviceMajor:0 DeviceMinor:400 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-430 DeviceMajor:0 DeviceMinor:430 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-242 DeviceMajor:0 DeviceMinor:242 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0b0122f6-d51a-4338-a631-056ec9311b69/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:254 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/35391db2-ab6e-4272-8580-9014e24f72c4/volumes/kubernetes.io~secret/tls-certificates DeviceMajor:0 DeviceMinor:292 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/805c87a3-63e9-4c13-9c90-c05f73282edc/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:327 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-363 DeviceMajor:0 DeviceMinor:363 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/efed176f934c493edca2ddc32acd11fa4382cd5265b4f1516ff814fbb11a2114/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a6cb025b09c74b2375547c39e2bcf11018d0fe067f47fa5bc687270d83d3c4d5/userdata/shm DeviceMajor:0 DeviceMinor:334 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e96638928e69534905932599d19846f2632b24018a9b807eb5d9c55f0b84bc3a/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4255f96b8c68d9e9d843b9faaca419295cf827087a981964731ed862bdadda7c/userdata/shm DeviceMajor:0 DeviceMinor:300 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-314 DeviceMajor:0 DeviceMinor:314 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/94b090bb2f81614395a6c92d20a127033288b6317a3746960c12ca9c06f937c8/userdata/shm DeviceMajor:0 DeviceMinor:341 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-393 DeviceMajor:0 DeviceMinor:393 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6149f269-49c2-4572-8f6c-fe01e9f0dcbf/volumes/kubernetes.io~projected/kube-api-access-ntxmg DeviceMajor:0 DeviceMinor:51 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/46168ed50cbd4d5ada910d8a21a69f2d8d81f28884def6ccc1f353dd392a4090/userdata/shm DeviceMajor:0 DeviceMinor:259 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/805c87a3-63e9-4c13-9c90-c05f73282edc/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:326 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-415 DeviceMajor:0 DeviceMinor:415 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-425 DeviceMajor:0 DeviceMinor:425 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4321e0ae-e339-40f2-875a-941b1183e766/volumes/kubernetes.io~projected/kube-api-access-2wlnj DeviceMajor:0 DeviceMinor:55 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0a973929be5cebc219b720e4cbc988c2101c4d3885f1252d602d91ece8d10c9b/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/46078cac-ae78-4ac7-bda3-b95913a478f8/volumes/kubernetes.io~projected/kube-api-access-djmg6 DeviceMajor:0 DeviceMinor:223 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7e9cccdd-6c18-4a60-837f-4907411a1c44/volumes/kubernetes.io~projected/kube-api-access-tmrx6 DeviceMajor:0 DeviceMinor:53 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-281 DeviceMajor:0 DeviceMinor:281 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a4e2086e72083ac7db69643e5261a496346ab4a08e69339adc535572a63c94f4/userdata/shm DeviceMajor:0 DeviceMinor:375 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4eff5ab2517b779c97c79777f957957fe446c9b09aba3853080f6cc1edf21822/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a8db8643-51bf-4bcf-b26d-bd937a67b5ed/volumes/kubernetes.io~projected/kube-api-access-4zn2l DeviceMajor:0 DeviceMinor:299 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1bd1166d-68e9-4035-b920-e42f5bf05d11/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:338 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-419 DeviceMajor:0 DeviceMinor:419 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d454e879-03a2-4fb9-8223-79a1aa9d85d2/volumes/kubernetes.io~projected/kube-api-access-hdlnm DeviceMajor:0 DeviceMinor:59 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/af0da37a-a117-49a6-a06e-6cab0e56a40c/volumes/kubernetes.io~projected/kube-api-access-hd22d DeviceMajor:0 DeviceMinor:197 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e4b4d6c6a74ba41ccfa800406dd6347bda703d9e8cde5971cf0cf9f4b1d1e118/userdata/shm DeviceMajor:0 DeviceMinor:354 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1d7634964deac3f00ec41b31563aff4f6799d2d66d08ad1fcfed920814d877a6/userdata/shm DeviceMajor:0 DeviceMinor:413 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0e24f687-655b-4edb-8901-35412d3fc79b/volumes/kubernetes.io~projected/kube-api-access-tgfc6 DeviceMajor:0 DeviceMinor:56 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e7b086d9337668f71e0f1d261a7126a33c053fb5fa12c69ccd7eb3548c960812/userdata/shm DeviceMajor:0 DeviceMinor:62 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0711f3d4e1d5868 MacAddress:8a:ea:80:d9:a7:e6 Speed:10000 Mtu:8901} {Name:1d7634964deac3f MacAddress:c6:a6:91:f0:9e:c3 Speed:10000 Mtu:8901} {Name:4255f96b8c68d9e MacAddress:ee:1b:32:d4:53:6e Speed:10000 Mtu:8901} {Name:46168ed50cbd4d5 MacAddress:66:9a:83:64:ed:be Speed:10000 Mtu:8901} {Name:4c6b70bec32c1ec MacAddress:4a:95:d5:25:06:1b Speed:10000 Mtu:8901} {Name:5db42fabf816db1 MacAddress:86:78:e6:8b:65:2c Speed:10000 Mtu:8901} {Name:77d30d727f12a09 MacAddress:16:40:e3:60:86:ad Speed:10000 Mtu:8901} {Name:78cfa2ceed0dfad MacAddress:c6:ee:da:1d:f9:49 Speed:10000 Mtu:8901} {Name:941920de309566d MacAddress:9e:76:83:d8:43:33 Speed:10000 Mtu:8901} {Name:94b090bb2f81614 MacAddress:9e:2f:64:ec:78:e5 Speed:10000 Mtu:8901} {Name:a4e2086e72083ac MacAddress:82:db:09:1b:20:4d Speed:10000 Mtu:8901} {Name:a6cb025b09c74b2 MacAddress:36:9e:fa:e3:41:3a Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:11:22:0f:a1:7b Speed:0 Mtu:9001} {Name:br-int MacAddress:ee:76:69:13:f6:ee Speed:0 Mtu:8901} {Name:d1f3f2d5ced6226 MacAddress:d6:4b:d9:33:8c:a9 Speed:10000 Mtu:8901} {Name:d69ee7d9a731d0d MacAddress:ce:35:47:f7:d0:92 Speed:10000 Mtu:8901} {Name:e4b4d6c6a74ba41 MacAddress:16:53:15:89:56:37 Speed:10000 Mtu:8901} {Name:efed176f934c493 MacAddress:fe:79:ad:65:5a:be Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:11:22:0f:a1:7b Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:ee:6f:1a:88:28:38 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:b2:f2:70:e2:43:8a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 01 05:03:42.004779 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.004656    5689 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 01 05:03:42.004779 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.004701    5689 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 01 05:03:42.005252 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.005039    5689 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 01 05:03:42.005252 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.005066    5689 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-92.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 01 05:03:42.005252 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.005246    5689 topology_manager.go:143] "Creating topology manager with none policy"
Nov 01 05:03:42.005400 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.005258    5689 container_manager_linux.go:311] "Creating device plugin manager"
Nov 01 05:03:42.005400 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.005273    5689 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 05:03:42.005400 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.005301    5689 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 05:03:42.005555 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.005440    5689 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 01 05:03:42.005685 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.005670    5689 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 01 05:03:42.005812 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.005800    5689 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 01 05:03:42.005868 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.005848    5689 kubelet.go:493] "Attempting to sync node with API server"
Nov 01 05:03:42.005868 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.005867    5689 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 01 05:03:42.005967 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.005881    5689 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 01 05:03:42.005967 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.005894    5689 kubelet.go:405] "Adding apiserver pod source"
Nov 01 05:03:42.005967 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.005907    5689 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 01 05:03:42.006750 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.006733    5689 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 01 05:03:42.006750 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.006748    5689 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 05:03:42.007239 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.007209    5689 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 01 05:03:42.020569 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.020553    5689 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 01 05:03:42.020569 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.020567    5689 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 05:03:42.020690 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.020660    5689 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 01 05:03:42.020890 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.020877    5689 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 01 05:03:42.020927 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.020908    5689 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 01 05:03:42.021011 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.021001    5689 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 01 05:03:42.021044 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.021018    5689 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 01 05:03:42.021081 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.021045    5689 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 01 05:03:42.021081 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.021052    5689 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 01 05:03:42.021081 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.021057    5689 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 01 05:03:42.021081 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.021062    5689 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 01 05:03:42.021081 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.021068    5689 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 01 05:03:42.021081 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.021074    5689 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 01 05:03:42.021081 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.021081    5689 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 01 05:03:42.021318 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.021086    5689 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 01 05:03:42.021318 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.021102    5689 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 01 05:03:42.021318 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.021112    5689 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 01 05:03:42.021318 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.021144    5689 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 01 05:03:42.021318 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.021151    5689 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 01 05:03:42.024523 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.024509    5689 server.go:1263] "Started kubelet"
Nov 01 05:03:42.025794 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.024625    5689 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 01 05:03:42.025794 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.024725    5689 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 01 05:03:42.025794 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.025104    5689 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 01 05:03:42.025794 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.025086    5689 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 01 05:03:42.025054 ip-10-0-129-92 systemd[1]: Started Kubernetes Kubelet.
Nov 01 05:03:42.027232 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.026900    5689 server.go:317] "Adding debug handlers to kubelet server"
Nov 01 05:03:42.029163 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.029144    5689 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 01 05:03:42.029965 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:03:42.029883    5689 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-92.us-west-1.compute.internal\" not found"
Nov 01 05:03:42.029965 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.029948    5689 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 01 05:03:42.029965 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.029956    5689 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 01 05:03:42.031845 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.029983    5689 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 01 05:03:42.031845 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.030838    5689 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 04:57:30 +0000 UTC" deadline="2028-07-25 05:21:51.60195179 +0000 UTC"
Nov 01 05:03:42.031845 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.030869    5689 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15168h18m9.571086309s"
Nov 01 05:03:42.031845 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.030884    5689 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 01 05:03:42.033136 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.033114    5689 factory.go:55] Registering systemd factory
Nov 01 05:03:42.033245 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.033141    5689 factory.go:223] Registration of the systemd container factory successfully
Nov 01 05:03:42.033381 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.033362    5689 factory.go:153] Registering CRI-O factory
Nov 01 05:03:42.033597 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.033387    5689 factory.go:223] Registration of the crio container factory successfully
Nov 01 05:03:42.033597 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.033445    5689 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 01 05:03:42.033597 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.033490    5689 factory.go:103] Registering Raw factory
Nov 01 05:03:42.033597 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.033511    5689 manager.go:1199] Started watching for new ooms in manager
Nov 01 05:03:42.033874 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.033851    5689 manager.go:319] Starting recovery of all containers
Nov 01 05:03:42.034148 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.034096    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a877f895-a4cf-404d-87e4-440d4b73a870" volumeName="kubernetes.io/projected/a877f895-a4cf-404d-87e4-440d4b73a870-kube-api-access-xc8bg" seLinuxMountContext=""
Nov 01 05:03:42.034224 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.034152    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cca47ebc-569a-4558-af41-5e3755605b9d" volumeName="kubernetes.io/configmap/cca47ebc-569a-4558-af41-5e3755605b9d-konnectivity-ca" seLinuxMountContext=""
Nov 01 05:03:42.034224 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.034167    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6149f269-49c2-4572-8f6c-fe01e9f0dcbf" volumeName="kubernetes.io/projected/6149f269-49c2-4572-8f6c-fe01e9f0dcbf-kube-api-access-ntxmg" seLinuxMountContext=""
Nov 01 05:03:42.034224 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.034178    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80c10aac-5175-46d6-a18b-109169a2461b" volumeName="kubernetes.io/empty-dir/80c10aac-5175-46d6-a18b-109169a2461b-tmp" seLinuxMountContext=""
Nov 01 05:03:42.034224 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.034191    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a9bb6df5-0993-4640-81cc-52426b72fb3e" volumeName="kubernetes.io/projected/a9bb6df5-0993-4640-81cc-52426b72fb3e-kube-api-access-n95qs" seLinuxMountContext=""
Nov 01 05:03:42.034224 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.034202    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca43c15d-cefb-4e78-93aa-a98573da4feb" volumeName="kubernetes.io/configmap/ca43c15d-cefb-4e78-93aa-a98573da4feb-service-ca" seLinuxMountContext=""
Nov 01 05:03:42.034224 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.034212    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca43c15d-cefb-4e78-93aa-a98573da4feb" volumeName="kubernetes.io/projected/ca43c15d-cefb-4e78-93aa-a98573da4feb-kube-api-access-wgwkh" seLinuxMountContext=""
Nov 01 05:03:42.034224 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.034223    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e9cccdd-6c18-4a60-837f-4907411a1c44" volumeName="kubernetes.io/configmap/7e9cccdd-6c18-4a60-837f-4907411a1c44-env-overrides" seLinuxMountContext=""
Nov 01 05:03:42.035017 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.034237    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8fba6aca-0a63-4924-89b2-87575d844ce6" volumeName="kubernetes.io/configmap/8fba6aca-0a63-4924-89b2-87575d844ce6-kube-rbac-proxy-cm" seLinuxMountContext=""
Nov 01 05:03:42.035017 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.034250    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1bd1166d-68e9-4035-b920-e42f5bf05d11" volumeName="kubernetes.io/configmap/1bd1166d-68e9-4035-b920-e42f5bf05d11-trusted-ca-bundle" seLinuxMountContext=""
Nov 01 05:03:42.035017 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.034263    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1bd1166d-68e9-4035-b920-e42f5bf05d11" volumeName="kubernetes.io/configmap/1bd1166d-68e9-4035-b920-e42f5bf05d11-console-config" seLinuxMountContext=""
Nov 01 05:03:42.035017 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.034274    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4321e0ae-e339-40f2-875a-941b1183e766" volumeName="kubernetes.io/secret/4321e0ae-e339-40f2-875a-941b1183e766-metrics-certs" seLinuxMountContext=""
Nov 01 05:03:42.035017 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.034284    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f4aae57-6429-4f9e-aaa5-c700bc26b4f5" volumeName="kubernetes.io/configmap/6f4aae57-6429-4f9e-aaa5-c700bc26b4f5-service-ca-bundle" seLinuxMountContext=""
Nov 01 05:03:42.035017 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.034298    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="972b129f-0166-4705-a71c-09ee0e7a91b9" volumeName="kubernetes.io/configmap/972b129f-0166-4705-a71c-09ee0e7a91b9-cni-binary-copy" seLinuxMountContext=""
Nov 01 05:03:42.035017 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.034312    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35391db2-ab6e-4272-8580-9014e24f72c4" volumeName="kubernetes.io/secret/35391db2-ab6e-4272-8580-9014e24f72c4-tls-certificates" seLinuxMountContext=""
Nov 01 05:03:42.035017 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.034324    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80c10aac-5175-46d6-a18b-109169a2461b" volumeName="kubernetes.io/secret/80c10aac-5175-46d6-a18b-109169a2461b-secret-thanos-querier-tls" seLinuxMountContext=""
Nov 01 05:03:42.035017 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.034336    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="972b129f-0166-4705-a71c-09ee0e7a91b9" volumeName="kubernetes.io/configmap/972b129f-0166-4705-a71c-09ee0e7a91b9-multus-daemon-config" seLinuxMountContext=""
Nov 01 05:03:42.035017 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.034362    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af0da37a-a117-49a6-a06e-6cab0e56a40c" volumeName="kubernetes.io/secret/af0da37a-a117-49a6-a06e-6cab0e56a40c-cert" seLinuxMountContext=""
Nov 01 05:03:42.035017 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.034370    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e856df5a-5c5e-4421-ae7d-80a760bc72e2" volumeName="kubernetes.io/secret/e856df5a-5c5e-4421-ae7d-80a760bc72e2-original-pull-secret" seLinuxMountContext=""
Nov 01 05:03:42.035017 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.034378    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b0122f6-d51a-4338-a631-056ec9311b69" volumeName="kubernetes.io/configmap/0b0122f6-d51a-4338-a631-056ec9311b69-config-volume" seLinuxMountContext=""
Nov 01 05:03:42.035017 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.034387    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1bd1166d-68e9-4035-b920-e42f5bf05d11" volumeName="kubernetes.io/configmap/1bd1166d-68e9-4035-b920-e42f5bf05d11-service-ca" seLinuxMountContext=""
Nov 01 05:03:42.035017 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.034399    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="805c87a3-63e9-4c13-9c90-c05f73282edc" volumeName="kubernetes.io/empty-dir/805c87a3-63e9-4c13-9c90-c05f73282edc-volume-directive-shadow" seLinuxMountContext=""
Nov 01 05:03:42.035017 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.034409    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="805c87a3-63e9-4c13-9c90-c05f73282edc" volumeName="kubernetes.io/projected/805c87a3-63e9-4c13-9c90-c05f73282edc-kube-api-access-hvgxl" seLinuxMountContext=""
Nov 01 05:03:42.035017 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.034416    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a9bb6df5-0993-4640-81cc-52426b72fb3e" volumeName="kubernetes.io/configmap/a9bb6df5-0993-4640-81cc-52426b72fb3e-cni-binary-copy" seLinuxMountContext=""
Nov 01 05:03:42.035017 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.034424    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b0122f6-d51a-4338-a631-056ec9311b69" volumeName="kubernetes.io/projected/0b0122f6-d51a-4338-a631-056ec9311b69-kube-api-access-9njs4" seLinuxMountContext=""
Nov 01 05:03:42.035017 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.034443    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6" volumeName="kubernetes.io/secret/0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6-secret-metrics-server-tls" seLinuxMountContext=""
Nov 01 05:03:42.035017 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.034471    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="46078cac-ae78-4ac7-bda3-b95913a478f8" volumeName="kubernetes.io/configmap/46078cac-ae78-4ac7-bda3-b95913a478f8-signing-cabundle" seLinuxMountContext=""
Nov 01 05:03:42.035017 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.034505    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e9cccdd-6c18-4a60-837f-4907411a1c44" volumeName="kubernetes.io/configmap/7e9cccdd-6c18-4a60-837f-4907411a1c44-ovnkube-script-lib" seLinuxMountContext=""
Nov 01 05:03:42.035017 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.034522    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9752ded2-725f-4035-b374-602e4b4eb576" volumeName="kubernetes.io/empty-dir/9752ded2-725f-4035-b374-602e4b4eb576-etc-tuned" seLinuxMountContext=""
Nov 01 05:03:42.035017 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.034534    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a9bb6df5-0993-4640-81cc-52426b72fb3e" volumeName="kubernetes.io/configmap/a9bb6df5-0993-4640-81cc-52426b72fb3e-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 01 05:03:42.035017 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.034546    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca43c15d-cefb-4e78-93aa-a98573da4feb" volumeName="kubernetes.io/secret/ca43c15d-cefb-4e78-93aa-a98573da4feb-console-serving-cert" seLinuxMountContext=""
Nov 01 05:03:42.035017 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.034580    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca43c15d-cefb-4e78-93aa-a98573da4feb" volumeName="kubernetes.io/secret/ca43c15d-cefb-4e78-93aa-a98573da4feb-console-oauth-config" seLinuxMountContext=""
Nov 01 05:03:42.035017 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.034592    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4321e0ae-e339-40f2-875a-941b1183e766" volumeName="kubernetes.io/projected/4321e0ae-e339-40f2-875a-941b1183e766-kube-api-access-2wlnj" seLinuxMountContext=""
Nov 01 05:03:42.035017 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.034604    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="46078cac-ae78-4ac7-bda3-b95913a478f8" volumeName="kubernetes.io/secret/46078cac-ae78-4ac7-bda3-b95913a478f8-signing-key" seLinuxMountContext=""
Nov 01 05:03:42.035017 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.034618    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e9cccdd-6c18-4a60-837f-4907411a1c44" volumeName="kubernetes.io/secret/7e9cccdd-6c18-4a60-837f-4907411a1c44-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 01 05:03:42.035017 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.034652    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8fba6aca-0a63-4924-89b2-87575d844ce6" volumeName="kubernetes.io/empty-dir/8fba6aca-0a63-4924-89b2-87575d844ce6-data-volume" seLinuxMountContext=""
Nov 01 05:03:42.035017 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.034667    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14" volumeName="kubernetes.io/projected/9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14-kube-api-access-r5zxr" seLinuxMountContext=""
Nov 01 05:03:42.035017 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.034679    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bcc64278-da5e-4525-90bf-abddde870d70" volumeName="kubernetes.io/projected/bcc64278-da5e-4525-90bf-abddde870d70-kube-api-access-ffjr4" seLinuxMountContext=""
Nov 01 05:03:42.035017 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.034705    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca43c15d-cefb-4e78-93aa-a98573da4feb" volumeName="kubernetes.io/configmap/ca43c15d-cefb-4e78-93aa-a98573da4feb-console-config" seLinuxMountContext=""
Nov 01 05:03:42.035017 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.034733    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14" volumeName="kubernetes.io/secret/9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14-node-exporter-tls" seLinuxMountContext=""
Nov 01 05:03:42.035017 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.034744    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8db8643-51bf-4bcf-b26d-bd937a67b5ed" volumeName="kubernetes.io/projected/a8db8643-51bf-4bcf-b26d-bd937a67b5ed-kube-api-access-4zn2l" seLinuxMountContext=""
Nov 01 05:03:42.035017 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.034752    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca43c15d-cefb-4e78-93aa-a98573da4feb" volumeName="kubernetes.io/empty-dir/ca43c15d-cefb-4e78-93aa-a98573da4feb-tmp" seLinuxMountContext=""
Nov 01 05:03:42.035017 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.034760    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1bd1166d-68e9-4035-b920-e42f5bf05d11" volumeName="kubernetes.io/projected/1bd1166d-68e9-4035-b920-e42f5bf05d11-kube-api-access-7npk7" seLinuxMountContext=""
Nov 01 05:03:42.035017 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.034767    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80c10aac-5175-46d6-a18b-109169a2461b" volumeName="kubernetes.io/configmap/80c10aac-5175-46d6-a18b-109169a2461b-metrics-client-ca" seLinuxMountContext=""
Nov 01 05:03:42.035017 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.034775    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14" volumeName="kubernetes.io/configmap/9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14-metrics-client-ca" seLinuxMountContext=""
Nov 01 05:03:42.035017 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.034783    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a877f895-a4cf-404d-87e4-440d4b73a870" volumeName="kubernetes.io/empty-dir/a877f895-a4cf-404d-87e4-440d4b73a870-tmp-dir" seLinuxMountContext=""
Nov 01 05:03:42.035017 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.034793    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca43c15d-cefb-4e78-93aa-a98573da4feb" volumeName="kubernetes.io/configmap/ca43c15d-cefb-4e78-93aa-a98573da4feb-oauth-serving-cert" seLinuxMountContext=""
Nov 01 05:03:42.035017 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.034800    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6" volumeName="kubernetes.io/secret/0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6-secret-metrics-server-client-certs" seLinuxMountContext=""
Nov 01 05:03:42.035017 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.034809    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1bd1166d-68e9-4035-b920-e42f5bf05d11" volumeName="kubernetes.io/configmap/1bd1166d-68e9-4035-b920-e42f5bf05d11-oauth-serving-cert" seLinuxMountContext=""
Nov 01 05:03:42.035017 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.034816    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9752ded2-725f-4035-b374-602e4b4eb576" volumeName="kubernetes.io/projected/9752ded2-725f-4035-b374-602e4b4eb576-kube-api-access-8rrgq" seLinuxMountContext=""
Nov 01 05:03:42.035017 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.034825    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8db8643-51bf-4bcf-b26d-bd937a67b5ed" volumeName="kubernetes.io/empty-dir/a8db8643-51bf-4bcf-b26d-bd937a67b5ed-tmp" seLinuxMountContext=""
Nov 01 05:03:42.035017 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.034833    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af0da37a-a117-49a6-a06e-6cab0e56a40c" volumeName="kubernetes.io/projected/af0da37a-a117-49a6-a06e-6cab0e56a40c-kube-api-access-hd22d" seLinuxMountContext=""
Nov 01 05:03:42.035017 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.034848    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80c10aac-5175-46d6-a18b-109169a2461b" volumeName="kubernetes.io/projected/80c10aac-5175-46d6-a18b-109169a2461b-kube-api-access-c7hhr" seLinuxMountContext=""
Nov 01 05:03:42.035017 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.034856    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6" volumeName="kubernetes.io/projected/0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6-kube-api-access-55zt7" seLinuxMountContext=""
Nov 01 05:03:42.035017 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.034869    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e24f687-655b-4edb-8901-35412d3fc79b" volumeName="kubernetes.io/projected/0e24f687-655b-4edb-8901-35412d3fc79b-kube-api-access-tgfc6" seLinuxMountContext=""
Nov 01 05:03:42.035017 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.034882    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f4aae57-6429-4f9e-aaa5-c700bc26b4f5" volumeName="kubernetes.io/empty-dir/6f4aae57-6429-4f9e-aaa5-c700bc26b4f5-archives-path" seLinuxMountContext=""
Nov 01 05:03:42.035017 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.034896    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f4aae57-6429-4f9e-aaa5-c700bc26b4f5" volumeName="kubernetes.io/projected/6f4aae57-6429-4f9e-aaa5-c700bc26b4f5-kube-api-access-248gt" seLinuxMountContext=""
Nov 01 05:03:42.035017 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.034904    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="972b129f-0166-4705-a71c-09ee0e7a91b9" volumeName="kubernetes.io/projected/972b129f-0166-4705-a71c-09ee0e7a91b9-kube-api-access-lv27z" seLinuxMountContext=""
Nov 01 05:03:42.035017 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.034912    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6" volumeName="kubernetes.io/configmap/0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6-metrics-server-audit-profiles" seLinuxMountContext=""
Nov 01 05:03:42.035017 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.034920    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6" volumeName="kubernetes.io/empty-dir/0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6-audit-log" seLinuxMountContext=""
Nov 01 05:03:42.035017 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.034927    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6149f269-49c2-4572-8f6c-fe01e9f0dcbf" volumeName="kubernetes.io/configmap/6149f269-49c2-4572-8f6c-fe01e9f0dcbf-iptables-alerter-script" seLinuxMountContext=""
Nov 01 05:03:42.035017 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.034935    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7cd84f11-44c9-4ec6-83c2-7db17ab9991c" volumeName="kubernetes.io/projected/7cd84f11-44c9-4ec6-83c2-7db17ab9991c-kube-api-access-8mg57" seLinuxMountContext=""
Nov 01 05:03:42.035017 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.034942    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80c10aac-5175-46d6-a18b-109169a2461b" volumeName="kubernetes.io/secret/80c10aac-5175-46d6-a18b-109169a2461b-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext=""
Nov 01 05:03:42.035017 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.034950    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6" volumeName="kubernetes.io/secret/0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6-client-ca-bundle" seLinuxMountContext=""
Nov 01 05:03:42.035017 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.034958    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1bd1166d-68e9-4035-b920-e42f5bf05d11" volumeName="kubernetes.io/secret/1bd1166d-68e9-4035-b920-e42f5bf05d11-console-serving-cert" seLinuxMountContext=""
Nov 01 05:03:42.035017 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.034965    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1bd1166d-68e9-4035-b920-e42f5bf05d11" volumeName="kubernetes.io/secret/1bd1166d-68e9-4035-b920-e42f5bf05d11-console-oauth-config" seLinuxMountContext=""
Nov 01 05:03:42.035017 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.034974    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e9cccdd-6c18-4a60-837f-4907411a1c44" volumeName="kubernetes.io/configmap/7e9cccdd-6c18-4a60-837f-4907411a1c44-ovnkube-config" seLinuxMountContext=""
Nov 01 05:03:42.035017 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.034982    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="805c87a3-63e9-4c13-9c90-c05f73282edc" volumeName="kubernetes.io/configmap/805c87a3-63e9-4c13-9c90-c05f73282edc-metrics-client-ca" seLinuxMountContext=""
Nov 01 05:03:42.035017 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.035004    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="805c87a3-63e9-4c13-9c90-c05f73282edc" volumeName="kubernetes.io/secret/805c87a3-63e9-4c13-9c90-c05f73282edc-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 01 05:03:42.035017 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.035012    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80c10aac-5175-46d6-a18b-109169a2461b" volumeName="kubernetes.io/secret/80c10aac-5175-46d6-a18b-109169a2461b-secret-grpc-tls" seLinuxMountContext=""
Nov 01 05:03:42.035017 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.035020    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6" volumeName="kubernetes.io/configmap/0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Nov 01 05:03:42.035017 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.035028    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f4aae57-6429-4f9e-aaa5-c700bc26b4f5" volumeName="kubernetes.io/configmap/6f4aae57-6429-4f9e-aaa5-c700bc26b4f5-trusted-ca-bundle" seLinuxMountContext=""
Nov 01 05:03:42.035017 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.035035    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="805c87a3-63e9-4c13-9c90-c05f73282edc" volumeName="kubernetes.io/configmap/805c87a3-63e9-4c13-9c90-c05f73282edc-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext=""
Nov 01 05:03:42.035017 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.035043    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="805c87a3-63e9-4c13-9c90-c05f73282edc" volumeName="kubernetes.io/secret/805c87a3-63e9-4c13-9c90-c05f73282edc-kube-state-metrics-tls" seLinuxMountContext=""
Nov 01 05:03:42.035017 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.035051    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8fba6aca-0a63-4924-89b2-87575d844ce6" volumeName="kubernetes.io/projected/8fba6aca-0a63-4924-89b2-87575d844ce6-kube-api-access-s6nj8" seLinuxMountContext=""
Nov 01 05:03:42.035017 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.035059    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14" volumeName="kubernetes.io/empty-dir/9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14-node-exporter-textfile" seLinuxMountContext=""
Nov 01 05:03:42.035017 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.035066    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e9cccdd-6c18-4a60-837f-4907411a1c44" volumeName="kubernetes.io/projected/7e9cccdd-6c18-4a60-837f-4907411a1c44-kube-api-access-tmrx6" seLinuxMountContext=""
Nov 01 05:03:42.035017 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.035075    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80c10aac-5175-46d6-a18b-109169a2461b" volumeName="kubernetes.io/secret/80c10aac-5175-46d6-a18b-109169a2461b-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext=""
Nov 01 05:03:42.038159 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.035083    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a9bb6df5-0993-4640-81cc-52426b72fb3e" volumeName="kubernetes.io/configmap/a9bb6df5-0993-4640-81cc-52426b72fb3e-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 01 05:03:42.038159 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.035114    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cca47ebc-569a-4558-af41-5e3755605b9d" volumeName="kubernetes.io/secret/cca47ebc-569a-4558-af41-5e3755605b9d-agent-certs" seLinuxMountContext=""
Nov 01 05:03:42.038159 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.035123    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1bd1166d-68e9-4035-b920-e42f5bf05d11" volumeName="kubernetes.io/empty-dir/1bd1166d-68e9-4035-b920-e42f5bf05d11-tmp" seLinuxMountContext=""
Nov 01 05:03:42.038159 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.035130    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="46078cac-ae78-4ac7-bda3-b95913a478f8" volumeName="kubernetes.io/projected/46078cac-ae78-4ac7-bda3-b95913a478f8-kube-api-access-djmg6" seLinuxMountContext=""
Nov 01 05:03:42.038159 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.035138    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80c10aac-5175-46d6-a18b-109169a2461b" volumeName="kubernetes.io/secret/80c10aac-5175-46d6-a18b-109169a2461b-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext=""
Nov 01 05:03:42.038159 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.035146    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8fba6aca-0a63-4924-89b2-87575d844ce6" volumeName="kubernetes.io/secret/8fba6aca-0a63-4924-89b2-87575d844ce6-insights-runtime-extractor-tls" seLinuxMountContext=""
Nov 01 05:03:42.038159 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.035154    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14" volumeName="kubernetes.io/secret/9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 01 05:03:42.038159 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.035162    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d454e879-03a2-4fb9-8223-79a1aa9d85d2" volumeName="kubernetes.io/configmap/d454e879-03a2-4fb9-8223-79a1aa9d85d2-serviceca" seLinuxMountContext=""
Nov 01 05:03:42.038159 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.035170    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d454e879-03a2-4fb9-8223-79a1aa9d85d2" volumeName="kubernetes.io/projected/d454e879-03a2-4fb9-8223-79a1aa9d85d2-kube-api-access-hdlnm" seLinuxMountContext=""
Nov 01 05:03:42.038159 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.035181    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b0122f6-d51a-4338-a631-056ec9311b69" volumeName="kubernetes.io/empty-dir/0b0122f6-d51a-4338-a631-056ec9311b69-tmp-dir" seLinuxMountContext=""
Nov 01 05:03:42.038159 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.035193    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b0122f6-d51a-4338-a631-056ec9311b69" volumeName="kubernetes.io/secret/0b0122f6-d51a-4338-a631-056ec9311b69-metrics-tls" seLinuxMountContext=""
Nov 01 05:03:42.038159 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.035200    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80c10aac-5175-46d6-a18b-109169a2461b" volumeName="kubernetes.io/secret/80c10aac-5175-46d6-a18b-109169a2461b-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext=""
Nov 01 05:03:42.038159 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.035208    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9752ded2-725f-4035-b374-602e4b4eb576" volumeName="kubernetes.io/empty-dir/9752ded2-725f-4035-b374-602e4b4eb576-tmp" seLinuxMountContext=""
Nov 01 05:03:42.038159 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.035215    5689 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14" volumeName="kubernetes.io/configmap/9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Nov 01 05:03:42.038159 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.035222    5689 reconstruct.go:97] "Volume reconstruction finished"
Nov 01 05:03:42.038159 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.035227    5689 reconciler.go:29] "Reconciler: start to sync state"
Nov 01 05:03:42.045533 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.045490    5689 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 01 05:03:42.059994 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.059965    5689 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 01 05:03:42.059994 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.059993    5689 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 01 05:03:42.060135 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.060016    5689 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 01 05:03:42.060135 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:03:42.060055    5689 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 01 05:03:42.072010 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.071982    5689 generic.go:356] "Generic (PLEG): container finished" podID="9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14" containerID="8e68c281df88a0c78cd55be82559c64946ed15163a4298d5f7fdae9720f71258" exitCode=0
Nov 01 05:03:42.082526 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.082500    5689 generic.go:356] "Generic (PLEG): container finished" podID="a9bb6df5-0993-4640-81cc-52426b72fb3e" containerID="dee5a815f5eaa4269c1bf5d730227d7a3b7f6f9ff38eeb573337ac08fd896dac" exitCode=0
Nov 01 05:03:42.082526 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.082524    5689 generic.go:356] "Generic (PLEG): container finished" podID="a9bb6df5-0993-4640-81cc-52426b72fb3e" containerID="478db2d69005981efb029034aec26cb2146045e035d1f79216434b4306c98e2e" exitCode=0
Nov 01 05:03:42.082677 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.082534    5689 generic.go:356] "Generic (PLEG): container finished" podID="a9bb6df5-0993-4640-81cc-52426b72fb3e" containerID="c6e7595015fd306e4bcd4b8c1de3981d99de3144748471519ad41938bb8ec6cd" exitCode=0
Nov 01 05:03:42.082677 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.082543    5689 generic.go:356] "Generic (PLEG): container finished" podID="a9bb6df5-0993-4640-81cc-52426b72fb3e" containerID="1e20a3e84f8ad12701497600c153da060cdd75e5a8e95c24363a5a7531206de5" exitCode=0
Nov 01 05:03:42.082677 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.082549    5689 generic.go:356] "Generic (PLEG): container finished" podID="a9bb6df5-0993-4640-81cc-52426b72fb3e" containerID="a7368cd51823aeba5ed2d567748e96aea78316e7f23b8bad5753e8d444b65622" exitCode=0
Nov 01 05:03:42.082677 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.082556    5689 generic.go:356] "Generic (PLEG): container finished" podID="a9bb6df5-0993-4640-81cc-52426b72fb3e" containerID="3e79c22b5e5c930ffb647b4821134e4ce2e2a3d76b0f1199fcb6ed2ca19648a2" exitCode=0
Nov 01 05:03:42.112475 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.112423    5689 generic.go:356] "Generic (PLEG): container finished" podID="99a2e3eae38c794f2284a61da069e044" containerID="0f8c39ab12f2e6999375d4135fb54dc15e3a611b01e27d18067ee26e6ca2db1c" exitCode=0
Nov 01 05:03:42.129983 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:03:42.129959    5689 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-92.us-west-1.compute.internal\" not found"
Nov 01 05:03:42.146660 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.146604    5689 manager.go:324] Recovery completed
Nov 01 05:03:42.161150 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:03:42.161114    5689 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Nov 01 05:03:42.167730 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.167713    5689 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 05:03:42.169585 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.169562    5689 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-92.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 05:03:42.169679 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.169595    5689 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-92.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 05:03:42.169679 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.169607    5689 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-92.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 05:03:42.172294 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.172274    5689 cpu_manager.go:226] "Starting" policy="none"
Nov 01 05:03:42.172294 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.172287    5689 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 01 05:03:42.172408 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.172296    5689 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 05:03:42.172408 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.172303    5689 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 01 05:03:42.172563 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.172544    5689 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 01 05:03:42.172612 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.172566    5689 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 01 05:03:42.172612 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.172585    5689 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 01 05:03:42.172612 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.172594    5689 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 01 05:03:42.172612 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.172602    5689 policy_none.go:50] "Start"
Nov 01 05:03:42.172723 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.172615    5689 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 01 05:03:42.172723 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.172630    5689 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 01 05:03:42.173219 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.173205    5689 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-92.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 05:03:42.173257 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.173231    5689 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-92.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 05:03:42.173257 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.173241    5689 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-92.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 05:03:42.173527 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.173513    5689 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 01 05:03:42.173571 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.173530    5689 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 01 05:03:42.173571 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.173537    5689 policy_none.go:44] "Start"
Nov 01 05:03:42.178070 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.178045    5689 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 01 05:03:42.180615 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.180600    5689 manager.go:341] "Starting Device Plugin manager"
Nov 01 05:03:42.180697 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:03:42.180625    5689 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 01 05:03:42.180697 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.180634    5689 server.go:87] "Starting device plugin registration server"
Nov 01 05:03:42.180873 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.180863    5689 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 01 05:03:42.180911 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.180875    5689 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 01 05:03:42.180987 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.180970    5689 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 01 05:03:42.181103 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.181064    5689 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 01 05:03:42.181209 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.181117    5689 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 01 05:03:42.181209 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.181125    5689 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 01 05:03:42.181962 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.181179    5689 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T05:03:42.181101136Z","UUID":"369ab168-cb68-4621-98c3-2464898e3002","Handler":null,"Name":"","Endpoint":""}
Nov 01 05:03:42.182852 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:03:42.182823    5689 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 01 05:03:42.182950 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:03:42.182861    5689 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-92.us-west-1.compute.internal\" not found"
Nov 01 05:03:42.183246 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.183230    5689 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 01 05:03:42.183321 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.183252    5689 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 01 05:03:42.217347 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.217323    5689 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 01 05:03:42.220414 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.220395    5689 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 05:03:42.220553 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.220419    5689 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 05:03:42.221013 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.220998    5689 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 05:03:42.281679 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.281657    5689 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 05:03:42.282603 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.282588    5689 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-92.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 05:03:42.282676 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.282616    5689 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-92.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 05:03:42.282676 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.282628    5689 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-92.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 05:03:42.282731 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.282676    5689 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-129-92.us-west-1.compute.internal"
Nov 01 05:03:42.316144 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.316122    5689 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 05:03:42.361670 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.361632    5689 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-129-92.us-west-1.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-92.us-west-1.compute.internal"]
Nov 01 05:03:42.362285 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.362272    5689 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-92.us-west-1.compute.internal"
Nov 01 05:03:42.362324 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.362290    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-92.us-west-1.compute.internal" event={"ID":"99a2e3eae38c794f2284a61da069e044","Type":"ContainerStarted","Data":"572fd80986c4e0d7538a35b388de3d4d19093ff7b75b47ae0da8d8cd5d73b317"}
Nov 01 05:03:42.362353 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.362329    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-92.us-west-1.compute.internal" event={"ID":"99a2e3eae38c794f2284a61da069e044","Type":"ContainerDied","Data":"0f8c39ab12f2e6999375d4135fb54dc15e3a611b01e27d18067ee26e6ca2db1c"}
Nov 01 05:03:42.362353 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.362346    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-92.us-west-1.compute.internal" event={"ID":"99a2e3eae38c794f2284a61da069e044","Type":"ContainerStarted","Data":"e96638928e69534905932599d19846f2632b24018a9b807eb5d9c55f0b84bc3a"}
Nov 01 05:03:42.362444 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.362358    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-92.us-west-1.compute.internal" event={"ID":"0742a37cb8250b02dcdb2d2301dda7a1","Type":"ContainerStarted","Data":"d516f6ed608b9ff59d17f0b67627c9272357bfe2fd97511db96b9548fa682736"}
Nov 01 05:03:42.362444 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.362369    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-92.us-west-1.compute.internal" event={"ID":"0742a37cb8250b02dcdb2d2301dda7a1","Type":"ContainerStarted","Data":"35f50f1e7eed82a9c567aa5d24724ef38198719bbf0c8a47d237d2da9df7aaa5"}
Nov 01 05:03:42.362544 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.362531    5689 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-92.us-west-1.compute.internal"
Nov 01 05:03:42.396009 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.395993    5689 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-129-92.us-west-1.compute.internal"
Nov 01 05:03:42.396100 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.396045    5689 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-129-92.us-west-1.compute.internal"
Nov 01 05:03:42.425194 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.425169    5689 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 05:03:42.425299 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:03:42.425216    5689 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-129-92.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-92.us-west-1.compute.internal"
Nov 01 05:03:42.428107 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.428092    5689 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 05:03:42.428156 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:03:42.428128    5689 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-129-92.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-92.us-west-1.compute.internal"
Nov 01 05:03:42.519565 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.519503    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0742a37cb8250b02dcdb2d2301dda7a1-config\") pod \"kube-apiserver-proxy-ip-10-0-129-92.us-west-1.compute.internal\" (UID: \"0742a37cb8250b02dcdb2d2301dda7a1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-92.us-west-1.compute.internal"
Nov 01 05:03:42.519722 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.519573    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/99a2e3eae38c794f2284a61da069e044-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-92.us-west-1.compute.internal\" (UID: \"99a2e3eae38c794f2284a61da069e044\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-92.us-west-1.compute.internal"
Nov 01 05:03:42.519722 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.519590    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/99a2e3eae38c794f2284a61da069e044-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-92.us-west-1.compute.internal\" (UID: \"99a2e3eae38c794f2284a61da069e044\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-92.us-west-1.compute.internal"
Nov 01 05:03:42.619893 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.619866    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/99a2e3eae38c794f2284a61da069e044-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-92.us-west-1.compute.internal\" (UID: \"99a2e3eae38c794f2284a61da069e044\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-92.us-west-1.compute.internal"
Nov 01 05:03:42.619893 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.619897    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0742a37cb8250b02dcdb2d2301dda7a1-config\") pod \"kube-apiserver-proxy-ip-10-0-129-92.us-west-1.compute.internal\" (UID: \"0742a37cb8250b02dcdb2d2301dda7a1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-92.us-west-1.compute.internal"
Nov 01 05:03:42.620071 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.619915    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/99a2e3eae38c794f2284a61da069e044-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-92.us-west-1.compute.internal\" (UID: \"99a2e3eae38c794f2284a61da069e044\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-92.us-west-1.compute.internal"
Nov 01 05:03:42.620071 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.619955    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/99a2e3eae38c794f2284a61da069e044-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-92.us-west-1.compute.internal\" (UID: \"99a2e3eae38c794f2284a61da069e044\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-92.us-west-1.compute.internal"
Nov 01 05:03:42.620071 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.619985    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/99a2e3eae38c794f2284a61da069e044-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-92.us-west-1.compute.internal\" (UID: \"99a2e3eae38c794f2284a61da069e044\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-92.us-west-1.compute.internal"
Nov 01 05:03:42.620071 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:42.619985    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0742a37cb8250b02dcdb2d2301dda7a1-config\") pod \"kube-apiserver-proxy-ip-10-0-129-92.us-west-1.compute.internal\" (UID: \"0742a37cb8250b02dcdb2d2301dda7a1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-92.us-west-1.compute.internal"
Nov 01 05:03:43.006726 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.006696    5689 apiserver.go:52] "Watching apiserver"
Nov 01 05:03:43.109647 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.109623    5689 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 01 05:03:43.110287 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.110262    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-kkng7","openshift-network-operator/iptables-alerter-d8jfj","kube-system/global-pull-secret-syncer-9pcxf","openshift-multus/network-metrics-daemon-267rv","openshift-ovn-kubernetes/ovnkube-node-2982h","openshift-service-ca/service-ca-5fffc97bdf-pwtc2","kube-system/konnectivity-agent-hzr4h","openshift-monitoring/kube-state-metrics-5dc8d7d89b-5hwhj","openshift-multus/multus-additional-cni-plugins-r4nh2","kube-system/kube-apiserver-proxy-ip-10-0-129-92.us-west-1.compute.internal","openshift-monitoring/metrics-server-7b6db58fc-994xl","openshift-network-diagnostics/network-check-target-sr4n8","openshift-console/console-76cf6f4bb6-dwvbm","openshift-console/downloads-6dd789c8b7-x7wfp","openshift-dns/dns-default-g89wb","openshift-cluster-node-tuning-operator/tuned-zm57v","openshift-image-registry/node-ca-vrbks","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-92.us-west-1.compute.internal","openshift-ingress-canary/ingress-canary-ljs77","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6kb4c","openshift-console/console-645d98cc97-fdph2","openshift-monitoring/node-exporter-vh7rl","openshift-insights/periodic-gathering-t7btw-l6t25","openshift-kube-storage-version-migrator/migrator-66df5d496b-jn2tf","openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-fbbdp","openshift-monitoring/thanos-querier-77ddfd4d86-btjkv","openshift-multus/multus-5cv94","openshift-dns/node-resolver-ddc5d"]
Nov 01 05:03:43.130256 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.130235    5689 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-92.us-west-1.compute.internal"
Nov 01 05:03:43.130402 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.130389    5689 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-92.us-west-1.compute.internal"
Nov 01 05:03:43.133082 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.133067    5689 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 01 05:03:43.164200 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.164168    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 01 05:03:43.164200 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.164169    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-q6tm4\""
Nov 01 05:03:43.164376 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.164230    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 01 05:03:43.164376 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.164238    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 01 05:03:43.164376 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.164281    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 01 05:03:43.164376 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.164294    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 01 05:03:43.164376 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.164297    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4vjvv\""
Nov 01 05:03:43.164376 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.164305    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 01 05:03:43.164376 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.164367    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 01 05:03:43.164944 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.164927    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 01 05:03:43.165021 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.165003    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 01 05:03:43.165021 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.165012    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 01 05:03:43.165171 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.165156    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 01 05:03:43.165244 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.165185    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 01 05:03:43.165244 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.165205    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 01 05:03:43.165244 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.165204    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 01 05:03:43.165244 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.165222    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-fbhl7\""
Nov 01 05:03:43.165244 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.165229    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 01 05:03:43.165405 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.165265    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-g7ktw\""
Nov 01 05:03:43.165556 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.165540    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 01 05:03:43.165763 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.165749    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6k7br\""
Nov 01 05:03:43.165828 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.165775    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-b9csn\""
Nov 01 05:03:43.165828 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.165778    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 01 05:03:43.165828 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.165804    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 01 05:03:43.165828 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.165805    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 01 05:03:43.165964 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.165891    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 01 05:03:43.166517 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.166499    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 01 05:03:43.166573 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.166535    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 01 05:03:43.166834 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.166818    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-nkwq4\""
Nov 01 05:03:43.166870 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.166821    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-tw68j\""
Nov 01 05:03:43.166870 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.166828    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 01 05:03:43.167279 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.167267    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-7kc5j\""
Nov 01 05:03:43.169089 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.169078    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 01 05:03:43.169164 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.169151    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-gnn2d\""
Nov 01 05:03:43.169380 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.169368    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-fhbr8\""
Nov 01 05:03:43.169685 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.169673    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6ndpw\""
Nov 01 05:03:43.169853 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.169841    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 01 05:03:43.170180 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.170168    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 01 05:03:43.170505 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.170485    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 01 05:03:43.170505 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.170489    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 01 05:03:43.170665 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.170503    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 01 05:03:43.170665 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.170523    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 01 05:03:43.170665 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.170572    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 01 05:03:43.170665 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.170575    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 01 05:03:43.214761 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.214739    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 01 05:03:43.214930 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.214917    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 01 05:03:43.215294 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.215277    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Nov 01 05:03:43.215344 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.215283    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Nov 01 05:03:43.215344 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.215334    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 01 05:03:43.215445 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.215405    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 01 05:03:43.215607 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.215565    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 01 05:03:43.215693 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.215630    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 01 05:03:43.215693 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.215679    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Nov 01 05:03:43.215693 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.215685    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Nov 01 05:03:43.215832 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.215686    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 01 05:03:43.215832 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.215741    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Nov 01 05:03:43.215832 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.215762    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 01 05:03:43.215832 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.215799    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 01 05:03:43.215832 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.215810    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Nov 01 05:03:43.215832 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.215821    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 01 05:03:43.216073 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.215840    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 01 05:03:43.216073 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.215944    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 01 05:03:43.216073 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.215955    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Nov 01 05:03:43.216421 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.216399    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 01 05:03:43.216544 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.216525    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 01 05:03:43.218826 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.218804    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Nov 01 05:03:43.218920 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.218903    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 01 05:03:43.218980 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.218934    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Nov 01 05:03:43.219988 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.219969    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 01 05:03:43.222376 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.222352    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/6f4aae57-6429-4f9e-aaa5-c700bc26b4f5-archives-path\") pod \"periodic-gathering-t7btw-l6t25\" (UID: \"6f4aae57-6429-4f9e-aaa5-c700bc26b4f5\") " pod="openshift-insights/periodic-gathering-t7btw-l6t25"
Nov 01 05:03:43.222488 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.222385    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14-metrics-client-ca\") pod \"node-exporter-vh7rl\" (UID: \"9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14\") " pod="openshift-monitoring/node-exporter-vh7rl"
Nov 01 05:03:43.222488 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.222400    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0b0122f6-d51a-4338-a631-056ec9311b69-config-volume\") pod \"dns-default-g89wb\" (UID: \"0b0122f6-d51a-4338-a631-056ec9311b69\") " pod="openshift-dns/dns-default-g89wb"
Nov 01 05:03:43.222488 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.222418    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9752ded2-725f-4035-b374-602e4b4eb576-etc-modprobe-d\") pod \"tuned-zm57v\" (UID: \"9752ded2-725f-4035-b374-602e4b4eb576\") " pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:03:43.222629 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.222498    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-host-cni-netd\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:03:43.222629 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.222536    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0e24f687-655b-4edb-8901-35412d3fc79b-device-dir\") pod \"aws-ebs-csi-driver-node-6kb4c\" (UID: \"0e24f687-655b-4edb-8901-35412d3fc79b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6kb4c"
Nov 01 05:03:43.222629 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.222541    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/6f4aae57-6429-4f9e-aaa5-c700bc26b4f5-archives-path\") pod \"periodic-gathering-t7btw-l6t25\" (UID: \"6f4aae57-6429-4f9e-aaa5-c700bc26b4f5\") " pod="openshift-insights/periodic-gathering-t7btw-l6t25"
Nov 01 05:03:43.222629 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.222568    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ca43c15d-cefb-4e78-93aa-a98573da4feb-oauth-serving-cert\") pod \"console-645d98cc97-fdph2\" (UID: \"ca43c15d-cefb-4e78-93aa-a98573da4feb\") " pod="openshift-console/console-645d98cc97-fdph2"
Nov 01 05:03:43.222629 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.222593    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14-sys\") pod \"node-exporter-vh7rl\" (UID: \"9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14\") " pod="openshift-monitoring/node-exporter-vh7rl"
Nov 01 05:03:43.222629 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.222625    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0b0122f6-d51a-4338-a631-056ec9311b69-metrics-tls\") pod \"dns-default-g89wb\" (UID: \"0b0122f6-d51a-4338-a631-056ec9311b69\") " pod="openshift-dns/dns-default-g89wb"
Nov 01 05:03:43.222919 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.222654    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a9bb6df5-0993-4640-81cc-52426b72fb3e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-r4nh2\" (UID: \"a9bb6df5-0993-4640-81cc-52426b72fb3e\") " pod="openshift-multus/multus-additional-cni-plugins-r4nh2"
Nov 01 05:03:43.222919 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.222666    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0b0122f6-d51a-4338-a631-056ec9311b69-config-volume\") pod \"dns-default-g89wb\" (UID: \"0b0122f6-d51a-4338-a631-056ec9311b69\") " pod="openshift-dns/dns-default-g89wb"
Nov 01 05:03:43.222919 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.222680    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgwkh\" (UniqueName: \"kubernetes.io/projected/ca43c15d-cefb-4e78-93aa-a98573da4feb-kube-api-access-wgwkh\") pod \"console-645d98cc97-fdph2\" (UID: \"ca43c15d-cefb-4e78-93aa-a98573da4feb\") " pod="openshift-console/console-645d98cc97-fdph2"
Nov 01 05:03:43.222919 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.222656    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14-metrics-client-ca\") pod \"node-exporter-vh7rl\" (UID: \"9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14\") " pod="openshift-monitoring/node-exporter-vh7rl"
Nov 01 05:03:43.222919 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.222706    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-systemd-units\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:03:43.222919 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.222741    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/805c87a3-63e9-4c13-9c90-c05f73282edc-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-5hwhj\" (UID: \"805c87a3-63e9-4c13-9c90-c05f73282edc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5hwhj"
Nov 01 05:03:43.222919 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.222760    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/972b129f-0166-4705-a71c-09ee0e7a91b9-host-var-lib-cni-bin\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:03:43.222919 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.222775    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:03:43.222919 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.222773    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ca43c15d-cefb-4e78-93aa-a98573da4feb-oauth-serving-cert\") pod \"console-645d98cc97-fdph2\" (UID: \"ca43c15d-cefb-4e78-93aa-a98573da4feb\") " pod="openshift-console/console-645d98cc97-fdph2"
Nov 01 05:03:43.222919 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.222839    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6-secret-metrics-server-tls\") pod \"metrics-server-7b6db58fc-994xl\" (UID: \"0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6\") " pod="openshift-monitoring/metrics-server-7b6db58fc-994xl"
Nov 01 05:03:43.222919 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.222867    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0b0122f6-d51a-4338-a631-056ec9311b69-metrics-tls\") pod \"dns-default-g89wb\" (UID: \"0b0122f6-d51a-4338-a631-056ec9311b69\") " pod="openshift-dns/dns-default-g89wb"
Nov 01 05:03:43.222919 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.222870    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4321e0ae-e339-40f2-875a-941b1183e766-metrics-certs\") pod \"network-metrics-daemon-267rv\" (UID: \"4321e0ae-e339-40f2-875a-941b1183e766\") " pod="openshift-multus/network-metrics-daemon-267rv"
Nov 01 05:03:43.223440 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.222927    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a9bb6df5-0993-4640-81cc-52426b72fb3e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-r4nh2\" (UID: \"a9bb6df5-0993-4640-81cc-52426b72fb3e\") " pod="openshift-multus/multus-additional-cni-plugins-r4nh2"
Nov 01 05:03:43.223440 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.222951    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/80c10aac-5175-46d6-a18b-109169a2461b-metrics-client-ca\") pod \"thanos-querier-77ddfd4d86-btjkv\" (UID: \"80c10aac-5175-46d6-a18b-109169a2461b\") " pod="openshift-monitoring/thanos-querier-77ddfd4d86-btjkv"
Nov 01 05:03:43.223440 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.222978    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/972b129f-0166-4705-a71c-09ee0e7a91b9-cnibin\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:03:43.223440 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.222987    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/805c87a3-63e9-4c13-9c90-c05f73282edc-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-5hwhj\" (UID: \"805c87a3-63e9-4c13-9c90-c05f73282edc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5hwhj"
Nov 01 05:03:43.223440 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.223034    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1bd1166d-68e9-4035-b920-e42f5bf05d11-console-serving-cert\") pod \"console-76cf6f4bb6-dwvbm\" (UID: \"1bd1166d-68e9-4035-b920-e42f5bf05d11\") " pod="openshift-console/console-76cf6f4bb6-dwvbm"
Nov 01 05:03:43.223440 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.223072    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4321e0ae-e339-40f2-875a-941b1183e766-metrics-certs\") pod \"network-metrics-daemon-267rv\" (UID: \"4321e0ae-e339-40f2-875a-941b1183e766\") " pod="openshift-multus/network-metrics-daemon-267rv"
Nov 01 05:03:43.223440 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.223074    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cca47ebc-569a-4558-af41-5e3755605b9d-agent-certs\") pod \"konnectivity-agent-hzr4h\" (UID: \"cca47ebc-569a-4558-af41-5e3755605b9d\") " pod="kube-system/konnectivity-agent-hzr4h"
Nov 01 05:03:43.223440 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.223115    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/80c10aac-5175-46d6-a18b-109169a2461b-metrics-client-ca\") pod \"thanos-querier-77ddfd4d86-btjkv\" (UID: \"80c10aac-5175-46d6-a18b-109169a2461b\") " pod="openshift-monitoring/thanos-querier-77ddfd4d86-btjkv"
Nov 01 05:03:43.223440 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.223119    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n95qs\" (UniqueName: \"kubernetes.io/projected/a9bb6df5-0993-4640-81cc-52426b72fb3e-kube-api-access-n95qs\") pod \"multus-additional-cni-plugins-r4nh2\" (UID: \"a9bb6df5-0993-4640-81cc-52426b72fb3e\") " pod="openshift-multus/multus-additional-cni-plugins-r4nh2"
Nov 01 05:03:43.223440 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.223153    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/af0da37a-a117-49a6-a06e-6cab0e56a40c-cert\") pod \"ingress-canary-ljs77\" (UID: \"af0da37a-a117-49a6-a06e-6cab0e56a40c\") " pod="openshift-ingress-canary/ingress-canary-ljs77"
Nov 01 05:03:43.223440 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.223170    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-host-slash\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:03:43.223440 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.223189    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-var-lib-openvswitch\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:03:43.223440 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.223254    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-run-ovn\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:03:43.223440 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.223257    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cca47ebc-569a-4558-af41-5e3755605b9d-agent-certs\") pod \"konnectivity-agent-hzr4h\" (UID: \"cca47ebc-569a-4558-af41-5e3755605b9d\") " pod="kube-system/konnectivity-agent-hzr4h"
Nov 01 05:03:43.223440 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.223289    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7hhr\" (UniqueName: \"kubernetes.io/projected/80c10aac-5175-46d6-a18b-109169a2461b-kube-api-access-c7hhr\") pod \"thanos-querier-77ddfd4d86-btjkv\" (UID: \"80c10aac-5175-46d6-a18b-109169a2461b\") " pod="openshift-monitoring/thanos-querier-77ddfd4d86-btjkv"
Nov 01 05:03:43.223440 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.223323    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a9bb6df5-0993-4640-81cc-52426b72fb3e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-r4nh2\" (UID: \"a9bb6df5-0993-4640-81cc-52426b72fb3e\") " pod="openshift-multus/multus-additional-cni-plugins-r4nh2"
Nov 01 05:03:43.223440 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.223339    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e856df5a-5c5e-4421-ae7d-80a760bc72e2-original-pull-secret\") pod \"global-pull-secret-syncer-9pcxf\" (UID: \"e856df5a-5c5e-4421-ae7d-80a760bc72e2\") " pod="kube-system/global-pull-secret-syncer-9pcxf"
Nov 01 05:03:43.223440 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.223367    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdlnm\" (UniqueName: \"kubernetes.io/projected/d454e879-03a2-4fb9-8223-79a1aa9d85d2-kube-api-access-hdlnm\") pod \"node-ca-vrbks\" (UID: \"d454e879-03a2-4fb9-8223-79a1aa9d85d2\") " pod="openshift-image-registry/node-ca-vrbks"
Nov 01 05:03:43.224252 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.223478    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6149f269-49c2-4572-8f6c-fe01e9f0dcbf-iptables-alerter-script\") pod \"iptables-alerter-d8jfj\" (UID: \"6149f269-49c2-4572-8f6c-fe01e9f0dcbf\") " pod="openshift-network-operator/iptables-alerter-d8jfj"
Nov 01 05:03:43.224252 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.223512    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvgxl\" (UniqueName: \"kubernetes.io/projected/805c87a3-63e9-4c13-9c90-c05f73282edc-kube-api-access-hvgxl\") pod \"kube-state-metrics-5dc8d7d89b-5hwhj\" (UID: \"805c87a3-63e9-4c13-9c90-c05f73282edc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5hwhj"
Nov 01 05:03:43.224252 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.223523    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e856df5a-5c5e-4421-ae7d-80a760bc72e2-original-pull-secret\") pod \"global-pull-secret-syncer-9pcxf\" (UID: \"e856df5a-5c5e-4421-ae7d-80a760bc72e2\") " pod="kube-system/global-pull-secret-syncer-9pcxf"
Nov 01 05:03:43.224252 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.223547    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9752ded2-725f-4035-b374-602e4b4eb576-var-lib-kubelet\") pod \"tuned-zm57v\" (UID: \"9752ded2-725f-4035-b374-602e4b4eb576\") " pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:03:43.224252 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.223582    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1bd1166d-68e9-4035-b920-e42f5bf05d11-tmp\") pod \"console-76cf6f4bb6-dwvbm\" (UID: \"1bd1166d-68e9-4035-b920-e42f5bf05d11\") " pod="openshift-console/console-76cf6f4bb6-dwvbm"
Nov 01 05:03:43.224252 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.223605    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e856df5a-5c5e-4421-ae7d-80a760bc72e2-kubelet-config\") pod \"global-pull-secret-syncer-9pcxf\" (UID: \"e856df5a-5c5e-4421-ae7d-80a760bc72e2\") " pod="kube-system/global-pull-secret-syncer-9pcxf"
Nov 01 05:03:43.224252 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.223613    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a9bb6df5-0993-4640-81cc-52426b72fb3e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-r4nh2\" (UID: \"a9bb6df5-0993-4640-81cc-52426b72fb3e\") " pod="openshift-multus/multus-additional-cni-plugins-r4nh2"
Nov 01 05:03:43.224252 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.223634    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8fba6aca-0a63-4924-89b2-87575d844ce6-data-volume\") pod \"insights-runtime-extractor-kkng7\" (UID: \"8fba6aca-0a63-4924-89b2-87575d844ce6\") " pod="openshift-insights/insights-runtime-extractor-kkng7"
Nov 01 05:03:43.224252 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.223650    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1bd1166d-68e9-4035-b920-e42f5bf05d11-tmp\") pod \"console-76cf6f4bb6-dwvbm\" (UID: \"1bd1166d-68e9-4035-b920-e42f5bf05d11\") " pod="openshift-console/console-76cf6f4bb6-dwvbm"
Nov 01 05:03:43.224252 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.223672    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/805c87a3-63e9-4c13-9c90-c05f73282edc-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-5hwhj\" (UID: \"805c87a3-63e9-4c13-9c90-c05f73282edc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5hwhj"
Nov 01 05:03:43.224252 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.223722    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/80c10aac-5175-46d6-a18b-109169a2461b-secret-grpc-tls\") pod \"thanos-querier-77ddfd4d86-btjkv\" (UID: \"80c10aac-5175-46d6-a18b-109169a2461b\") " pod="openshift-monitoring/thanos-querier-77ddfd4d86-btjkv"
Nov 01 05:03:43.224252 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.223729    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8fba6aca-0a63-4924-89b2-87575d844ce6-data-volume\") pod \"insights-runtime-extractor-kkng7\" (UID: \"8fba6aca-0a63-4924-89b2-87575d844ce6\") " pod="openshift-insights/insights-runtime-extractor-kkng7"
Nov 01 05:03:43.224252 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.223736    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6149f269-49c2-4572-8f6c-fe01e9f0dcbf-iptables-alerter-script\") pod \"iptables-alerter-d8jfj\" (UID: \"6149f269-49c2-4572-8f6c-fe01e9f0dcbf\") " pod="openshift-network-operator/iptables-alerter-d8jfj"
Nov 01 05:03:43.224252 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.223765    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0e24f687-655b-4edb-8901-35412d3fc79b-sys-fs\") pod \"aws-ebs-csi-driver-node-6kb4c\" (UID: \"0e24f687-655b-4edb-8901-35412d3fc79b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6kb4c"
Nov 01 05:03:43.224252 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.223804    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d454e879-03a2-4fb9-8223-79a1aa9d85d2-serviceca\") pod \"node-ca-vrbks\" (UID: \"d454e879-03a2-4fb9-8223-79a1aa9d85d2\") " pod="openshift-image-registry/node-ca-vrbks"
Nov 01 05:03:43.224252 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.223829    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ca43c15d-cefb-4e78-93aa-a98573da4feb-console-config\") pod \"console-645d98cc97-fdph2\" (UID: \"ca43c15d-cefb-4e78-93aa-a98573da4feb\") " pod="openshift-console/console-645d98cc97-fdph2"
Nov 01 05:03:43.224252 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.223857    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/805c87a3-63e9-4c13-9c90-c05f73282edc-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-5hwhj\" (UID: \"805c87a3-63e9-4c13-9c90-c05f73282edc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5hwhj"
Nov 01 05:03:43.224252 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.223938    5689 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 05:03:43.224252 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.223969    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7e9cccdd-6c18-4a60-837f-4907411a1c44-ovn-node-metrics-cert\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:03:43.224252 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:03:43.224006    5689 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-129-92.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-92.us-west-1.compute.internal"
Nov 01 05:03:43.224252 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.223939    5689 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 05:03:43.224252 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:03:43.224076    5689 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-129-92.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-92.us-west-1.compute.internal"
Nov 01 05:03:43.224252 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.224009    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/972b129f-0166-4705-a71c-09ee0e7a91b9-multus-daemon-config\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:03:43.224252 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.224109    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ca43c15d-cefb-4e78-93aa-a98573da4feb-console-config\") pod \"console-645d98cc97-fdph2\" (UID: \"ca43c15d-cefb-4e78-93aa-a98573da4feb\") " pod="openshift-console/console-645d98cc97-fdph2"
Nov 01 05:03:43.224252 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.224130    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mg57\" (UniqueName: \"kubernetes.io/projected/7cd84f11-44c9-4ec6-83c2-7db17ab9991c-kube-api-access-8mg57\") pod \"migrator-66df5d496b-jn2tf\" (UID: \"7cd84f11-44c9-4ec6-83c2-7db17ab9991c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jn2tf"
Nov 01 05:03:43.224252 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.224158    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/805c87a3-63e9-4c13-9c90-c05f73282edc-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-5hwhj\" (UID: \"805c87a3-63e9-4c13-9c90-c05f73282edc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5hwhj"
Nov 01 05:03:43.224252 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.224193    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1bd1166d-68e9-4035-b920-e42f5bf05d11-oauth-serving-cert\") pod \"console-76cf6f4bb6-dwvbm\" (UID: \"1bd1166d-68e9-4035-b920-e42f5bf05d11\") " pod="openshift-console/console-76cf6f4bb6-dwvbm"
Nov 01 05:03:43.224252 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.224219    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6-audit-log\") pod \"metrics-server-7b6db58fc-994xl\" (UID: \"0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6\") " pod="openshift-monitoring/metrics-server-7b6db58fc-994xl"
Nov 01 05:03:43.224252 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.224195    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/972b129f-0166-4705-a71c-09ee0e7a91b9-multus-daemon-config\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:03:43.224252 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.224242    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/35391db2-ab6e-4272-8580-9014e24f72c4-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-fbbdp\" (UID: \"35391db2-ab6e-4272-8580-9014e24f72c4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-fbbdp"
Nov 01 05:03:43.224252 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.224237    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7e9cccdd-6c18-4a60-837f-4907411a1c44-ovn-node-metrics-cert\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:03:43.224252 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.224267    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd22d\" (UniqueName: \"kubernetes.io/projected/af0da37a-a117-49a6-a06e-6cab0e56a40c-kube-api-access-hd22d\") pod \"ingress-canary-ljs77\" (UID: \"af0da37a-a117-49a6-a06e-6cab0e56a40c\") " pod="openshift-ingress-canary/ingress-canary-ljs77"
Nov 01 05:03:43.225295 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.224347    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8fba6aca-0a63-4924-89b2-87575d844ce6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kkng7\" (UID: \"8fba6aca-0a63-4924-89b2-87575d844ce6\") " pod="openshift-insights/insights-runtime-extractor-kkng7"
Nov 01 05:03:43.225295 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.224374    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9752ded2-725f-4035-b374-602e4b4eb576-etc-systemd\") pod \"tuned-zm57v\" (UID: \"9752ded2-725f-4035-b374-602e4b4eb576\") " pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:03:43.225295 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.224386    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6-audit-log\") pod \"metrics-server-7b6db58fc-994xl\" (UID: \"0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6\") " pod="openshift-monitoring/metrics-server-7b6db58fc-994xl"
Nov 01 05:03:43.225295 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.224428    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9752ded2-725f-4035-b374-602e4b4eb576-run\") pod \"tuned-zm57v\" (UID: \"9752ded2-725f-4035-b374-602e4b4eb576\") " pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:03:43.225295 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.224444    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1bd1166d-68e9-4035-b920-e42f5bf05d11-oauth-serving-cert\") pod \"console-76cf6f4bb6-dwvbm\" (UID: \"1bd1166d-68e9-4035-b920-e42f5bf05d11\") " pod="openshift-console/console-76cf6f4bb6-dwvbm"
Nov 01 05:03:43.225295 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.224480    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9752ded2-725f-4035-b374-602e4b4eb576-lib-modules\") pod \"tuned-zm57v\" (UID: \"9752ded2-725f-4035-b374-602e4b4eb576\") " pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:03:43.225295 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.224478    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d454e879-03a2-4fb9-8223-79a1aa9d85d2-serviceca\") pod \"node-ca-vrbks\" (UID: \"d454e879-03a2-4fb9-8223-79a1aa9d85d2\") " pod="openshift-image-registry/node-ca-vrbks"
Nov 01 05:03:43.225295 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.224513    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/972b129f-0166-4705-a71c-09ee0e7a91b9-hostroot\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:03:43.225295 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.224544    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc8bg\" (UniqueName: \"kubernetes.io/projected/a877f895-a4cf-404d-87e4-440d4b73a870-kube-api-access-xc8bg\") pod \"node-resolver-ddc5d\" (UID: \"a877f895-a4cf-404d-87e4-440d4b73a870\") " pod="openshift-dns/node-resolver-ddc5d"
Nov 01 05:03:43.225295 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.224574    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffjr4\" (UniqueName: \"kubernetes.io/projected/bcc64278-da5e-4525-90bf-abddde870d70-kube-api-access-ffjr4\") pod \"network-check-target-sr4n8\" (UID: \"bcc64278-da5e-4525-90bf-abddde870d70\") " pod="openshift-network-diagnostics/network-check-target-sr4n8"
Nov 01 05:03:43.225295 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.224643    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/46078cac-ae78-4ac7-bda3-b95913a478f8-signing-cabundle\") pod \"service-ca-5fffc97bdf-pwtc2\" (UID: \"46078cac-ae78-4ac7-bda3-b95913a478f8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-pwtc2"
Nov 01 05:03:43.225295 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.224676    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/972b129f-0166-4705-a71c-09ee0e7a91b9-multus-socket-dir-parent\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:03:43.225295 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.224702    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/972b129f-0166-4705-a71c-09ee0e7a91b9-system-cni-dir\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:03:43.225295 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.224728    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv27z\" (UniqueName: \"kubernetes.io/projected/972b129f-0166-4705-a71c-09ee0e7a91b9-kube-api-access-lv27z\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:03:43.225295 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.224755    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14-node-exporter-textfile\") pod \"node-exporter-vh7rl\" (UID: \"9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14\") " pod="openshift-monitoring/node-exporter-vh7rl"
Nov 01 05:03:43.225295 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.224798    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/46078cac-ae78-4ac7-bda3-b95913a478f8-signing-cabundle\") pod \"service-ca-5fffc97bdf-pwtc2\" (UID: \"46078cac-ae78-4ac7-bda3-b95913a478f8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-pwtc2"
Nov 01 05:03:43.225295 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.224898    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14-node-exporter-textfile\") pod \"node-exporter-vh7rl\" (UID: \"9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14\") " pod="openshift-monitoring/node-exporter-vh7rl"
Nov 01 05:03:43.225295 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.224923    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1bd1166d-68e9-4035-b920-e42f5bf05d11-console-config\") pod \"console-76cf6f4bb6-dwvbm\" (UID: \"1bd1166d-68e9-4035-b920-e42f5bf05d11\") " pod="openshift-console/console-76cf6f4bb6-dwvbm"
Nov 01 05:03:43.225295 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.224964    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wlnj\" (UniqueName: \"kubernetes.io/projected/4321e0ae-e339-40f2-875a-941b1183e766-kube-api-access-2wlnj\") pod \"network-metrics-daemon-267rv\" (UID: \"4321e0ae-e339-40f2-875a-941b1183e766\") " pod="openshift-multus/network-metrics-daemon-267rv"
Nov 01 05:03:43.225295 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.225048    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rrgq\" (UniqueName: \"kubernetes.io/projected/9752ded2-725f-4035-b374-602e4b4eb576-kube-api-access-8rrgq\") pod \"tuned-zm57v\" (UID: \"9752ded2-725f-4035-b374-602e4b4eb576\") " pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:03:43.225295 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.225072    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1bd1166d-68e9-4035-b920-e42f5bf05d11-console-config\") pod \"console-76cf6f4bb6-dwvbm\" (UID: \"1bd1166d-68e9-4035-b920-e42f5bf05d11\") " pod="openshift-console/console-76cf6f4bb6-dwvbm"
Nov 01 05:03:43.225295 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.225141    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/972b129f-0166-4705-a71c-09ee0e7a91b9-cni-binary-copy\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:03:43.225295 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.225239    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/972b129f-0166-4705-a71c-09ee0e7a91b9-host-run-netns\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:03:43.226234 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.225328    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f4aae57-6429-4f9e-aaa5-c700bc26b4f5-trusted-ca-bundle\") pod \"periodic-gathering-t7btw-l6t25\" (UID: \"6f4aae57-6429-4f9e-aaa5-c700bc26b4f5\") " pod="openshift-insights/periodic-gathering-t7btw-l6t25"
Nov 01 05:03:43.226234 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.225369    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-248gt\" (UniqueName: \"kubernetes.io/projected/6f4aae57-6429-4f9e-aaa5-c700bc26b4f5-kube-api-access-248gt\") pod \"periodic-gathering-t7btw-l6t25\" (UID: \"6f4aae57-6429-4f9e-aaa5-c700bc26b4f5\") " pod="openshift-insights/periodic-gathering-t7btw-l6t25"
Nov 01 05:03:43.226234 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.225397    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/972b129f-0166-4705-a71c-09ee0e7a91b9-cni-binary-copy\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:03:43.226234 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.225436    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14-root\") pod \"node-exporter-vh7rl\" (UID: \"9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14\") " pod="openshift-monitoring/node-exporter-vh7rl"
Nov 01 05:03:43.226234 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.225515    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-run-systemd\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:03:43.226234 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.225566    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-etc-openvswitch\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:03:43.226234 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.225588    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9752ded2-725f-4035-b374-602e4b4eb576-etc-sysconfig\") pod \"tuned-zm57v\" (UID: \"9752ded2-725f-4035-b374-602e4b4eb576\") " pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:03:43.226234 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.225604    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a9bb6df5-0993-4640-81cc-52426b72fb3e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-r4nh2\" (UID: \"a9bb6df5-0993-4640-81cc-52426b72fb3e\") " pod="openshift-multus/multus-additional-cni-plugins-r4nh2"
Nov 01 05:03:43.226234 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.225622    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f4aae57-6429-4f9e-aaa5-c700bc26b4f5-trusted-ca-bundle\") pod \"periodic-gathering-t7btw-l6t25\" (UID: \"6f4aae57-6429-4f9e-aaa5-c700bc26b4f5\") " pod="openshift-insights/periodic-gathering-t7btw-l6t25"
Nov 01 05:03:43.226234 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.225655    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0e24f687-655b-4edb-8901-35412d3fc79b-etc-selinux\") pod \"aws-ebs-csi-driver-node-6kb4c\" (UID: \"0e24f687-655b-4edb-8901-35412d3fc79b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6kb4c"
Nov 01 05:03:43.226234 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.225679    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d454e879-03a2-4fb9-8223-79a1aa9d85d2-host\") pod \"node-ca-vrbks\" (UID: \"d454e879-03a2-4fb9-8223-79a1aa9d85d2\") " pod="openshift-image-registry/node-ca-vrbks"
Nov 01 05:03:43.226234 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.225698    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8fba6aca-0a63-4924-89b2-87575d844ce6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kkng7\" (UID: \"8fba6aca-0a63-4924-89b2-87575d844ce6\") " pod="openshift-insights/insights-runtime-extractor-kkng7"
Nov 01 05:03:43.226234 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.225754    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/80c10aac-5175-46d6-a18b-109169a2461b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-77ddfd4d86-btjkv\" (UID: \"80c10aac-5175-46d6-a18b-109169a2461b\") " pod="openshift-monitoring/thanos-querier-77ddfd4d86-btjkv"
Nov 01 05:03:43.226234 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.225781    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9752ded2-725f-4035-b374-602e4b4eb576-etc-kubernetes\") pod \"tuned-zm57v\" (UID: \"9752ded2-725f-4035-b374-602e4b4eb576\") " pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:03:43.226234 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.225800    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a877f895-a4cf-404d-87e4-440d4b73a870-tmp-dir\") pod \"node-resolver-ddc5d\" (UID: \"a877f895-a4cf-404d-87e4-440d4b73a870\") " pod="openshift-dns/node-resolver-ddc5d"
Nov 01 05:03:43.226234 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.225830    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8fba6aca-0a63-4924-89b2-87575d844ce6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kkng7\" (UID: \"8fba6aca-0a63-4924-89b2-87575d844ce6\") " pod="openshift-insights/insights-runtime-extractor-kkng7"
Nov 01 05:03:43.226234 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.225845    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7e9cccdd-6c18-4a60-837f-4907411a1c44-env-overrides\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:03:43.226234 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.225873    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7b6db58fc-994xl\" (UID: \"0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6\") " pod="openshift-monitoring/metrics-server-7b6db58fc-994xl"
Nov 01 05:03:43.226234 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.225911    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a877f895-a4cf-404d-87e4-440d4b73a870-tmp-dir\") pod \"node-resolver-ddc5d\" (UID: \"a877f895-a4cf-404d-87e4-440d4b73a870\") " pod="openshift-dns/node-resolver-ddc5d"
Nov 01 05:03:43.226234 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.225955    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b0122f6-d51a-4338-a631-056ec9311b69-tmp-dir\") pod \"dns-default-g89wb\" (UID: \"0b0122f6-d51a-4338-a631-056ec9311b69\") " pod="openshift-dns/dns-default-g89wb"
Nov 01 05:03:43.226234 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.225966    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7e9cccdd-6c18-4a60-837f-4907411a1c44-env-overrides\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:03:43.226234 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.226023    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/46078cac-ae78-4ac7-bda3-b95913a478f8-signing-key\") pod \"service-ca-5fffc97bdf-pwtc2\" (UID: \"46078cac-ae78-4ac7-bda3-b95913a478f8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-pwtc2"
Nov 01 05:03:43.226234 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.226045    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1bd1166d-68e9-4035-b920-e42f5bf05d11-console-oauth-config\") pod \"console-76cf6f4bb6-dwvbm\" (UID: \"1bd1166d-68e9-4035-b920-e42f5bf05d11\") " pod="openshift-console/console-76cf6f4bb6-dwvbm"
Nov 01 05:03:43.226234 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.226053    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7b6db58fc-994xl\" (UID: \"0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6\") " pod="openshift-monitoring/metrics-server-7b6db58fc-994xl"
Nov 01 05:03:43.226234 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.226085    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/972b129f-0166-4705-a71c-09ee0e7a91b9-host-run-k8s-cni-cncf-io\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:03:43.226234 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.226119    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b0122f6-d51a-4338-a631-056ec9311b69-tmp-dir\") pod \"dns-default-g89wb\" (UID: \"0b0122f6-d51a-4338-a631-056ec9311b69\") " pod="openshift-dns/dns-default-g89wb"
Nov 01 05:03:43.226234 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.226144    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0e24f687-655b-4edb-8901-35412d3fc79b-socket-dir\") pod \"aws-ebs-csi-driver-node-6kb4c\" (UID: \"0e24f687-655b-4edb-8901-35412d3fc79b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6kb4c"
Nov 01 05:03:43.226234 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.226177    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/972b129f-0166-4705-a71c-09ee0e7a91b9-etc-kubernetes\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:03:43.226234 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.226206    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9752ded2-725f-4035-b374-602e4b4eb576-etc-tuned\") pod \"tuned-zm57v\" (UID: \"9752ded2-725f-4035-b374-602e4b4eb576\") " pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:03:43.226234 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.226221    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f4aae57-6429-4f9e-aaa5-c700bc26b4f5-service-ca-bundle\") pod \"periodic-gathering-t7btw-l6t25\" (UID: \"6f4aae57-6429-4f9e-aaa5-c700bc26b4f5\") " pod="openshift-insights/periodic-gathering-t7btw-l6t25"
Nov 01 05:03:43.226234 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.226248    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca43c15d-cefb-4e78-93aa-a98573da4feb-tmp\") pod \"console-645d98cc97-fdph2\" (UID: \"ca43c15d-cefb-4e78-93aa-a98573da4feb\") " pod="openshift-console/console-645d98cc97-fdph2"
Nov 01 05:03:43.227650 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.226290    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9752ded2-725f-4035-b374-602e4b4eb576-etc-tuned\") pod \"tuned-zm57v\" (UID: \"9752ded2-725f-4035-b374-602e4b4eb576\") " pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:03:43.227650 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.226306    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca43c15d-cefb-4e78-93aa-a98573da4feb-tmp\") pod \"console-645d98cc97-fdph2\" (UID: \"ca43c15d-cefb-4e78-93aa-a98573da4feb\") " pod="openshift-console/console-645d98cc97-fdph2"
Nov 01 05:03:43.227650 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.226318    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-run-openvswitch\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:03:43.227650 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.226337    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/805c87a3-63e9-4c13-9c90-c05f73282edc-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-5hwhj\" (UID: \"805c87a3-63e9-4c13-9c90-c05f73282edc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5hwhj"
Nov 01 05:03:43.227650 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.226357    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7npk7\" (UniqueName: \"kubernetes.io/projected/1bd1166d-68e9-4035-b920-e42f5bf05d11-kube-api-access-7npk7\") pod \"console-76cf6f4bb6-dwvbm\" (UID: \"1bd1166d-68e9-4035-b920-e42f5bf05d11\") " pod="openshift-console/console-76cf6f4bb6-dwvbm"
Nov 01 05:03:43.227650 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.226411    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6-secret-metrics-server-client-certs\") pod \"metrics-server-7b6db58fc-994xl\" (UID: \"0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6\") " pod="openshift-monitoring/metrics-server-7b6db58fc-994xl"
Nov 01 05:03:43.227650 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.226437    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9njs4\" (UniqueName: \"kubernetes.io/projected/0b0122f6-d51a-4338-a631-056ec9311b69-kube-api-access-9njs4\") pod \"dns-default-g89wb\" (UID: \"0b0122f6-d51a-4338-a631-056ec9311b69\") " pod="openshift-dns/dns-default-g89wb"
Nov 01 05:03:43.227650 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.226546    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/80c10aac-5175-46d6-a18b-109169a2461b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-77ddfd4d86-btjkv\" (UID: \"80c10aac-5175-46d6-a18b-109169a2461b\") " pod="openshift-monitoring/thanos-querier-77ddfd4d86-btjkv"
Nov 01 05:03:43.227650 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.226576    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14-node-exporter-accelerators-collector-config\") pod \"node-exporter-vh7rl\" (UID: \"9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14\") " pod="openshift-monitoring/node-exporter-vh7rl"
Nov 01 05:03:43.227650 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.226663    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5zxr\" (UniqueName: \"kubernetes.io/projected/9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14-kube-api-access-r5zxr\") pod \"node-exporter-vh7rl\" (UID: \"9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14\") " pod="openshift-monitoring/node-exporter-vh7rl"
Nov 01 05:03:43.227650 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.226757    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7e9cccdd-6c18-4a60-837f-4907411a1c44-ovnkube-script-lib\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:03:43.227650 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.226785    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14-node-exporter-accelerators-collector-config\") pod \"node-exporter-vh7rl\" (UID: \"9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14\") " pod="openshift-monitoring/node-exporter-vh7rl"
Nov 01 05:03:43.227650 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.226794    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/80c10aac-5175-46d6-a18b-109169a2461b-tmp\") pod \"thanos-querier-77ddfd4d86-btjkv\" (UID: \"80c10aac-5175-46d6-a18b-109169a2461b\") " pod="openshift-monitoring/thanos-querier-77ddfd4d86-btjkv"
Nov 01 05:03:43.227650 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.226820    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9752ded2-725f-4035-b374-602e4b4eb576-etc-sysctl-conf\") pod \"tuned-zm57v\" (UID: \"9752ded2-725f-4035-b374-602e4b4eb576\") " pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:03:43.227650 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.226837    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a9bb6df5-0993-4640-81cc-52426b72fb3e-cni-binary-copy\") pod \"multus-additional-cni-plugins-r4nh2\" (UID: \"a9bb6df5-0993-4640-81cc-52426b72fb3e\") " pod="openshift-multus/multus-additional-cni-plugins-r4nh2"
Nov 01 05:03:43.227650 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.226856    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/972b129f-0166-4705-a71c-09ee0e7a91b9-host-run-multus-certs\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:03:43.227650 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.226857    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f4aae57-6429-4f9e-aaa5-c700bc26b4f5-service-ca-bundle\") pod \"periodic-gathering-t7btw-l6t25\" (UID: \"6f4aae57-6429-4f9e-aaa5-c700bc26b4f5\") " pod="openshift-insights/periodic-gathering-t7btw-l6t25"
Nov 01 05:03:43.227650 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.226925    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8fba6aca-0a63-4924-89b2-87575d844ce6-crio-socket\") pod \"insights-runtime-extractor-kkng7\" (UID: \"8fba6aca-0a63-4924-89b2-87575d844ce6\") " pod="openshift-insights/insights-runtime-extractor-kkng7"
Nov 01 05:03:43.227650 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.226929    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/80c10aac-5175-46d6-a18b-109169a2461b-tmp\") pod \"thanos-querier-77ddfd4d86-btjkv\" (UID: \"80c10aac-5175-46d6-a18b-109169a2461b\") " pod="openshift-monitoring/thanos-querier-77ddfd4d86-btjkv"
Nov 01 05:03:43.227650 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.226949    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7e9cccdd-6c18-4a60-837f-4907411a1c44-ovnkube-script-lib\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:03:43.227650 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.226970    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgfc6\" (UniqueName: \"kubernetes.io/projected/0e24f687-655b-4edb-8901-35412d3fc79b-kube-api-access-tgfc6\") pod \"aws-ebs-csi-driver-node-6kb4c\" (UID: \"0e24f687-655b-4edb-8901-35412d3fc79b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6kb4c"
Nov 01 05:03:43.227650 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.226992    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/972b129f-0166-4705-a71c-09ee0e7a91b9-host-var-lib-cni-multus\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:03:43.227650 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.227011    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ca43c15d-cefb-4e78-93aa-a98573da4feb-service-ca\") pod \"console-645d98cc97-fdph2\" (UID: \"ca43c15d-cefb-4e78-93aa-a98573da4feb\") " pod="openshift-console/console-645d98cc97-fdph2"
Nov 01 05:03:43.227650 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.227023    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a9bb6df5-0993-4640-81cc-52426b72fb3e-cni-binary-copy\") pod \"multus-additional-cni-plugins-r4nh2\" (UID: \"a9bb6df5-0993-4640-81cc-52426b72fb3e\") " pod="openshift-multus/multus-additional-cni-plugins-r4nh2"
Nov 01 05:03:43.227650 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.227045    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14-node-exporter-tls\") pod \"node-exporter-vh7rl\" (UID: \"9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14\") " pod="openshift-monitoring/node-exporter-vh7rl"
Nov 01 05:03:43.227650 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.227149    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-host-kubelet\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:03:43.227650 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.227189    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-host-run-ovn-kubernetes\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:03:43.227650 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.227194    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ca43c15d-cefb-4e78-93aa-a98573da4feb-service-ca\") pod \"console-645d98cc97-fdph2\" (UID: \"ca43c15d-cefb-4e78-93aa-a98573da4feb\") " pod="openshift-console/console-645d98cc97-fdph2"
Nov 01 05:03:43.227650 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.227224    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-host-cni-bin\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:03:43.227650 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.227247    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/805c87a3-63e9-4c13-9c90-c05f73282edc-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-5hwhj\" (UID: \"805c87a3-63e9-4c13-9c90-c05f73282edc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5hwhj"
Nov 01 05:03:43.227650 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.227264    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-host-run-netns\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:03:43.227650 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.227292    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-node-log\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:03:43.227650 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.227308    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6-client-ca-bundle\") pod \"metrics-server-7b6db58fc-994xl\" (UID: \"0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6\") " pod="openshift-monitoring/metrics-server-7b6db58fc-994xl"
Nov 01 05:03:43.227650 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.227308    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/805c87a3-63e9-4c13-9c90-c05f73282edc-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-5hwhj\" (UID: \"805c87a3-63e9-4c13-9c90-c05f73282edc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5hwhj"
Nov 01 05:03:43.227650 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.227332    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9752ded2-725f-4035-b374-602e4b4eb576-etc-sysctl-d\") pod \"tuned-zm57v\" (UID: \"9752ded2-725f-4035-b374-602e4b4eb576\") " pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:03:43.227650 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.227349    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9752ded2-725f-4035-b374-602e4b4eb576-sys\") pod \"tuned-zm57v\" (UID: \"9752ded2-725f-4035-b374-602e4b4eb576\") " pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:03:43.227650 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.227399    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a9bb6df5-0993-4640-81cc-52426b72fb3e-system-cni-dir\") pod \"multus-additional-cni-plugins-r4nh2\" (UID: \"a9bb6df5-0993-4640-81cc-52426b72fb3e\") " pod="openshift-multus/multus-additional-cni-plugins-r4nh2"
Nov 01 05:03:43.227650 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.227433    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntxmg\" (UniqueName: \"kubernetes.io/projected/6149f269-49c2-4572-8f6c-fe01e9f0dcbf-kube-api-access-ntxmg\") pod \"iptables-alerter-d8jfj\" (UID: \"6149f269-49c2-4572-8f6c-fe01e9f0dcbf\") " pod="openshift-network-operator/iptables-alerter-d8jfj"
Nov 01 05:03:43.227650 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.227487    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a877f895-a4cf-404d-87e4-440d4b73a870-hosts-file\") pod \"node-resolver-ddc5d\" (UID: \"a877f895-a4cf-404d-87e4-440d4b73a870\") " pod="openshift-dns/node-resolver-ddc5d"
Nov 01 05:03:43.227650 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.227515    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ca43c15d-cefb-4e78-93aa-a98573da4feb-console-oauth-config\") pod \"console-645d98cc97-fdph2\" (UID: \"ca43c15d-cefb-4e78-93aa-a98573da4feb\") " pod="openshift-console/console-645d98cc97-fdph2"
Nov 01 05:03:43.227650 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.227542    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6-metrics-server-audit-profiles\") pod \"metrics-server-7b6db58fc-994xl\" (UID: \"0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6\") " pod="openshift-monitoring/metrics-server-7b6db58fc-994xl"
Nov 01 05:03:43.227650 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.227585    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/972b129f-0166-4705-a71c-09ee0e7a91b9-multus-conf-dir\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:03:43.227650 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.227653    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7e9cccdd-6c18-4a60-837f-4907411a1c44-ovnkube-config\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:03:43.227650 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.227677    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmrx6\" (UniqueName: \"kubernetes.io/projected/7e9cccdd-6c18-4a60-837f-4907411a1c44-kube-api-access-tmrx6\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:03:43.228956 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.227694    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/80c10aac-5175-46d6-a18b-109169a2461b-secret-thanos-querier-tls\") pod \"thanos-querier-77ddfd4d86-btjkv\" (UID: \"80c10aac-5175-46d6-a18b-109169a2461b\") " pod="openshift-monitoring/thanos-querier-77ddfd4d86-btjkv"
Nov 01 05:03:43.228956 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.227709    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a8db8643-51bf-4bcf-b26d-bd937a67b5ed-tmp\") pod \"downloads-6dd789c8b7-x7wfp\" (UID: \"a8db8643-51bf-4bcf-b26d-bd937a67b5ed\") " pod="openshift-console/downloads-6dd789c8b7-x7wfp"
Nov 01 05:03:43.228956 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.227724    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6149f269-49c2-4572-8f6c-fe01e9f0dcbf-host-slash\") pod \"iptables-alerter-d8jfj\" (UID: \"6149f269-49c2-4572-8f6c-fe01e9f0dcbf\") " pod="openshift-network-operator/iptables-alerter-d8jfj"
Nov 01 05:03:43.228956 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.227738    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9752ded2-725f-4035-b374-602e4b4eb576-tmp\") pod \"tuned-zm57v\" (UID: \"9752ded2-725f-4035-b374-602e4b4eb576\") " pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:03:43.228956 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.227760    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/972b129f-0166-4705-a71c-09ee0e7a91b9-multus-cni-dir\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:03:43.228956 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.227793    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ca43c15d-cefb-4e78-93aa-a98573da4feb-console-serving-cert\") pod \"console-645d98cc97-fdph2\" (UID: \"ca43c15d-cefb-4e78-93aa-a98573da4feb\") " pod="openshift-console/console-645d98cc97-fdph2"
Nov 01 05:03:43.228956 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.227806    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a8db8643-51bf-4bcf-b26d-bd937a67b5ed-tmp\") pod \"downloads-6dd789c8b7-x7wfp\" (UID: \"a8db8643-51bf-4bcf-b26d-bd937a67b5ed\") " pod="openshift-console/downloads-6dd789c8b7-x7wfp"
Nov 01 05:03:43.228956 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.227808    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6-metrics-server-audit-profiles\") pod \"metrics-server-7b6db58fc-994xl\" (UID: \"0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6\") " pod="openshift-monitoring/metrics-server-7b6db58fc-994xl"
Nov 01 05:03:43.228956 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.227834    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9752ded2-725f-4035-b374-602e4b4eb576-tmp\") pod \"tuned-zm57v\" (UID: \"9752ded2-725f-4035-b374-602e4b4eb576\") " pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:03:43.228956 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.227849    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7e9cccdd-6c18-4a60-837f-4907411a1c44-ovnkube-config\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:03:43.228956 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.227857    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cca47ebc-569a-4558-af41-5e3755605b9d-konnectivity-ca\") pod \"konnectivity-agent-hzr4h\" (UID: \"cca47ebc-569a-4558-af41-5e3755605b9d\") " pod="kube-system/konnectivity-agent-hzr4h"
Nov 01 05:03:43.228956 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.227892    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a9bb6df5-0993-4640-81cc-52426b72fb3e-cnibin\") pod \"multus-additional-cni-plugins-r4nh2\" (UID: \"a9bb6df5-0993-4640-81cc-52426b72fb3e\") " pod="openshift-multus/multus-additional-cni-plugins-r4nh2"
Nov 01 05:03:43.228956 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.227908    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0e24f687-655b-4edb-8901-35412d3fc79b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6kb4c\" (UID: \"0e24f687-655b-4edb-8901-35412d3fc79b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6kb4c"
Nov 01 05:03:43.228956 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.227930    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vh7rl\" (UID: \"9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14\") " pod="openshift-monitoring/node-exporter-vh7rl"
Nov 01 05:03:43.228956 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.227956    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55zt7\" (UniqueName: \"kubernetes.io/projected/0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6-kube-api-access-55zt7\") pod \"metrics-server-7b6db58fc-994xl\" (UID: \"0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6\") " pod="openshift-monitoring/metrics-server-7b6db58fc-994xl"
Nov 01 05:03:43.228956 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.227994    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0e24f687-655b-4edb-8901-35412d3fc79b-registration-dir\") pod \"aws-ebs-csi-driver-node-6kb4c\" (UID: \"0e24f687-655b-4edb-8901-35412d3fc79b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6kb4c"
Nov 01 05:03:43.228956 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.228021    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cca47ebc-569a-4558-af41-5e3755605b9d-konnectivity-ca\") pod \"konnectivity-agent-hzr4h\" (UID: \"cca47ebc-569a-4558-af41-5e3755605b9d\") " pod="kube-system/konnectivity-agent-hzr4h"
Nov 01 05:03:43.228956 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.228092    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/972b129f-0166-4705-a71c-09ee0e7a91b9-host-var-lib-kubelet\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:03:43.228956 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.228120    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1bd1166d-68e9-4035-b920-e42f5bf05d11-trusted-ca-bundle\") pod \"console-76cf6f4bb6-dwvbm\" (UID: \"1bd1166d-68e9-4035-b920-e42f5bf05d11\") " pod="openshift-console/console-76cf6f4bb6-dwvbm"
Nov 01 05:03:43.228956 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.228150    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djmg6\" (UniqueName: \"kubernetes.io/projected/46078cac-ae78-4ac7-bda3-b95913a478f8-kube-api-access-djmg6\") pod \"service-ca-5fffc97bdf-pwtc2\" (UID: \"46078cac-ae78-4ac7-bda3-b95913a478f8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-pwtc2"
Nov 01 05:03:43.228956 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.228185    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zn2l\" (UniqueName: \"kubernetes.io/projected/a8db8643-51bf-4bcf-b26d-bd937a67b5ed-kube-api-access-4zn2l\") pod \"downloads-6dd789c8b7-x7wfp\" (UID: \"a8db8643-51bf-4bcf-b26d-bd937a67b5ed\") " pod="openshift-console/downloads-6dd789c8b7-x7wfp"
Nov 01 05:03:43.228956 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.228213    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14-node-exporter-wtmp\") pod \"node-exporter-vh7rl\" (UID: \"9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14\") " pod="openshift-monitoring/node-exporter-vh7rl"
Nov 01 05:03:43.228956 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.228237    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-log-socket\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:03:43.228956 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.228262    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1bd1166d-68e9-4035-b920-e42f5bf05d11-service-ca\") pod \"console-76cf6f4bb6-dwvbm\" (UID: \"1bd1166d-68e9-4035-b920-e42f5bf05d11\") " pod="openshift-console/console-76cf6f4bb6-dwvbm"
Nov 01 05:03:43.228956 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.228287    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9752ded2-725f-4035-b374-602e4b4eb576-host\") pod \"tuned-zm57v\" (UID: \"9752ded2-725f-4035-b374-602e4b4eb576\") " pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:03:43.228956 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.228312    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e856df5a-5c5e-4421-ae7d-80a760bc72e2-dbus\") pod \"global-pull-secret-syncer-9pcxf\" (UID: \"e856df5a-5c5e-4421-ae7d-80a760bc72e2\") " pod="kube-system/global-pull-secret-syncer-9pcxf"
Nov 01 05:03:43.228956 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.228350    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6nj8\" (UniqueName: \"kubernetes.io/projected/8fba6aca-0a63-4924-89b2-87575d844ce6-kube-api-access-s6nj8\") pod \"insights-runtime-extractor-kkng7\" (UID: \"8fba6aca-0a63-4924-89b2-87575d844ce6\") " pod="openshift-insights/insights-runtime-extractor-kkng7"
Nov 01 05:03:43.228956 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.228377    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/80c10aac-5175-46d6-a18b-109169a2461b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-77ddfd4d86-btjkv\" (UID: \"80c10aac-5175-46d6-a18b-109169a2461b\") " pod="openshift-monitoring/thanos-querier-77ddfd4d86-btjkv"
Nov 01 05:03:43.228956 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.228402    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/80c10aac-5175-46d6-a18b-109169a2461b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-77ddfd4d86-btjkv\" (UID: \"80c10aac-5175-46d6-a18b-109169a2461b\") " pod="openshift-monitoring/thanos-querier-77ddfd4d86-btjkv"
Nov 01 05:03:43.228956 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.228404    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1bd1166d-68e9-4035-b920-e42f5bf05d11-trusted-ca-bundle\") pod \"console-76cf6f4bb6-dwvbm\" (UID: \"1bd1166d-68e9-4035-b920-e42f5bf05d11\") " pod="openshift-console/console-76cf6f4bb6-dwvbm"
Nov 01 05:03:43.228956 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.228801    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1bd1166d-68e9-4035-b920-e42f5bf05d11-service-ca\") pod \"console-76cf6f4bb6-dwvbm\" (UID: \"1bd1166d-68e9-4035-b920-e42f5bf05d11\") " pod="openshift-console/console-76cf6f4bb6-dwvbm"
Nov 01 05:03:43.249489 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.249471    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 01 05:03:43.264323 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.264242    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 01 05:03:43.264323 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.264278    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Nov 01 05:03:43.264323 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.264302    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-bp8rt\""
Nov 01 05:03:43.264323 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.264312    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 01 05:03:43.264614 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.264599    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-t567f\""
Nov 01 05:03:43.264654 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.264599    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 01 05:03:43.264720 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.264703    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8fba6aca-0a63-4924-89b2-87575d844ce6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kkng7\" (UID: \"8fba6aca-0a63-4924-89b2-87575d844ce6\") " pod="openshift-insights/insights-runtime-extractor-kkng7"
Nov 01 05:03:43.264951 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.264937    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Nov 01 05:03:43.264994 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.264948    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Nov 01 05:03:43.264994 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.264976    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-b2kibd754ifqu\""
Nov 01 05:03:43.265104 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.265090    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Nov 01 05:03:43.265156 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.265119    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-62qzp\""
Nov 01 05:03:43.265156 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.265128    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Nov 01 05:03:43.265156 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.265138    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-slwj6\""
Nov 01 05:03:43.265293 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.265170    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-btxwf\""
Nov 01 05:03:43.265293 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.265196    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Nov 01 05:03:43.265293 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.265206    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Nov 01 05:03:43.265293 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.265240    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Nov 01 05:03:43.265293 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.265247    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Nov 01 05:03:43.265293 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.265250    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Nov 01 05:03:43.265293 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.265258    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-sxmdz\""
Nov 01 05:03:43.265293 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.265213    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-jtxhq\""
Nov 01 05:03:43.265293 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.265288    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 01 05:03:43.265619 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.265297    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Nov 01 05:03:43.265619 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.265304    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-27c4k\""
Nov 01 05:03:43.265619 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.265309    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-fkkn4\""
Nov 01 05:03:43.265619 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.265313    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xvz28\""
Nov 01 05:03:43.265619 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.265321    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-b8lb60f3fpm0o\""
Nov 01 05:03:43.265619 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.265333    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Nov 01 05:03:43.265619 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.265352    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Nov 01 05:03:43.265619 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.265362    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-zqvfm\""
Nov 01 05:03:43.265619 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.265411    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-zskd9\""
Nov 01 05:03:43.265619 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.265432    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-7m49c\""
Nov 01 05:03:43.266619 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.266603    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/80c10aac-5175-46d6-a18b-109169a2461b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-77ddfd4d86-btjkv\" (UID: \"80c10aac-5175-46d6-a18b-109169a2461b\") " pod="openshift-monitoring/thanos-querier-77ddfd4d86-btjkv"
Nov 01 05:03:43.266681 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.266606    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/46078cac-ae78-4ac7-bda3-b95913a478f8-signing-key\") pod \"service-ca-5fffc97bdf-pwtc2\" (UID: \"46078cac-ae78-4ac7-bda3-b95913a478f8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-pwtc2"
Nov 01 05:03:43.266681 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.266630    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1bd1166d-68e9-4035-b920-e42f5bf05d11-console-oauth-config\") pod \"console-76cf6f4bb6-dwvbm\" (UID: \"1bd1166d-68e9-4035-b920-e42f5bf05d11\") " pod="openshift-console/console-76cf6f4bb6-dwvbm"
Nov 01 05:03:43.266681 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.266651    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/805c87a3-63e9-4c13-9c90-c05f73282edc-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-5hwhj\" (UID: \"805c87a3-63e9-4c13-9c90-c05f73282edc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5hwhj"
Nov 01 05:03:43.266681 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.266671    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6-secret-metrics-server-client-certs\") pod \"metrics-server-7b6db58fc-994xl\" (UID: \"0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6\") " pod="openshift-monitoring/metrics-server-7b6db58fc-994xl"
Nov 01 05:03:43.266803 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.266726    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/80c10aac-5175-46d6-a18b-109169a2461b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-77ddfd4d86-btjkv\" (UID: \"80c10aac-5175-46d6-a18b-109169a2461b\") " pod="openshift-monitoring/thanos-querier-77ddfd4d86-btjkv"
Nov 01 05:03:43.267801 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.267784    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14-node-exporter-tls\") pod \"node-exporter-vh7rl\" (UID: \"9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14\") " pod="openshift-monitoring/node-exporter-vh7rl"
Nov 01 05:03:43.267835 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.267788    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6-client-ca-bundle\") pod \"metrics-server-7b6db58fc-994xl\" (UID: \"0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6\") " pod="openshift-monitoring/metrics-server-7b6db58fc-994xl"
Nov 01 05:03:43.267835 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.267790    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ca43c15d-cefb-4e78-93aa-a98573da4feb-console-oauth-config\") pod \"console-645d98cc97-fdph2\" (UID: \"ca43c15d-cefb-4e78-93aa-a98573da4feb\") " pod="openshift-console/console-645d98cc97-fdph2"
Nov 01 05:03:43.267920 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.267907    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/80c10aac-5175-46d6-a18b-109169a2461b-secret-thanos-querier-tls\") pod \"thanos-querier-77ddfd4d86-btjkv\" (UID: \"80c10aac-5175-46d6-a18b-109169a2461b\") " pod="openshift-monitoring/thanos-querier-77ddfd4d86-btjkv"
Nov 01 05:03:43.268032 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.268020    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ca43c15d-cefb-4e78-93aa-a98573da4feb-console-serving-cert\") pod \"console-645d98cc97-fdph2\" (UID: \"ca43c15d-cefb-4e78-93aa-a98573da4feb\") " pod="openshift-console/console-645d98cc97-fdph2"
Nov 01 05:03:43.268935 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.268921    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vh7rl\" (UID: \"9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14\") " pod="openshift-monitoring/node-exporter-vh7rl"
Nov 01 05:03:43.269002 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.268990    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/80c10aac-5175-46d6-a18b-109169a2461b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-77ddfd4d86-btjkv\" (UID: \"80c10aac-5175-46d6-a18b-109169a2461b\") " pod="openshift-monitoring/thanos-querier-77ddfd4d86-btjkv"
Nov 01 05:03:43.269033 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.269013    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/80c10aac-5175-46d6-a18b-109169a2461b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-77ddfd4d86-btjkv\" (UID: \"80c10aac-5175-46d6-a18b-109169a2461b\") " pod="openshift-monitoring/thanos-querier-77ddfd4d86-btjkv"
Nov 01 05:03:43.273165 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.273151    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6-secret-metrics-server-tls\") pod \"metrics-server-7b6db58fc-994xl\" (UID: \"0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6\") " pod="openshift-monitoring/metrics-server-7b6db58fc-994xl"
Nov 01 05:03:43.273305 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.273294    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1bd1166d-68e9-4035-b920-e42f5bf05d11-console-serving-cert\") pod \"console-76cf6f4bb6-dwvbm\" (UID: \"1bd1166d-68e9-4035-b920-e42f5bf05d11\") " pod="openshift-console/console-76cf6f4bb6-dwvbm"
Nov 01 05:03:43.273365 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.273353    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/af0da37a-a117-49a6-a06e-6cab0e56a40c-cert\") pod \"ingress-canary-ljs77\" (UID: \"af0da37a-a117-49a6-a06e-6cab0e56a40c\") " pod="openshift-ingress-canary/ingress-canary-ljs77"
Nov 01 05:03:43.274896 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.274875    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/805c87a3-63e9-4c13-9c90-c05f73282edc-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-5hwhj\" (UID: \"805c87a3-63e9-4c13-9c90-c05f73282edc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5hwhj"
Nov 01 05:03:43.274952 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.274928    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/35391db2-ab6e-4272-8580-9014e24f72c4-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-fbbdp\" (UID: \"35391db2-ab6e-4272-8580-9014e24f72c4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-fbbdp"
Nov 01 05:03:43.274952 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.274945    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/80c10aac-5175-46d6-a18b-109169a2461b-secret-grpc-tls\") pod \"thanos-querier-77ddfd4d86-btjkv\" (UID: \"80c10aac-5175-46d6-a18b-109169a2461b\") " pod="openshift-monitoring/thanos-querier-77ddfd4d86-btjkv"
Nov 01 05:03:43.283384 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.283368    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv27z\" (UniqueName: \"kubernetes.io/projected/972b129f-0166-4705-a71c-09ee0e7a91b9-kube-api-access-lv27z\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:03:43.283384 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.283378    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgwkh\" (UniqueName: \"kubernetes.io/projected/ca43c15d-cefb-4e78-93aa-a98573da4feb-kube-api-access-wgwkh\") pod \"console-645d98cc97-fdph2\" (UID: \"ca43c15d-cefb-4e78-93aa-a98573da4feb\") " pod="openshift-console/console-645d98cc97-fdph2"
Nov 01 05:03:43.284095 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.284079    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdlnm\" (UniqueName: \"kubernetes.io/projected/d454e879-03a2-4fb9-8223-79a1aa9d85d2-kube-api-access-hdlnm\") pod \"node-ca-vrbks\" (UID: \"d454e879-03a2-4fb9-8223-79a1aa9d85d2\") " pod="openshift-image-registry/node-ca-vrbks"
Nov 01 05:03:43.284561 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.284543    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffjr4\" (UniqueName: \"kubernetes.io/projected/bcc64278-da5e-4525-90bf-abddde870d70-kube-api-access-ffjr4\") pod \"network-check-target-sr4n8\" (UID: \"bcc64278-da5e-4525-90bf-abddde870d70\") " pod="openshift-network-diagnostics/network-check-target-sr4n8"
Nov 01 05:03:43.291429 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.291411    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6nj8\" (UniqueName: \"kubernetes.io/projected/8fba6aca-0a63-4924-89b2-87575d844ce6-kube-api-access-s6nj8\") pod \"insights-runtime-extractor-kkng7\" (UID: \"8fba6aca-0a63-4924-89b2-87575d844ce6\") " pod="openshift-insights/insights-runtime-extractor-kkng7"
Nov 01 05:03:43.291429 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.291422    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc8bg\" (UniqueName: \"kubernetes.io/projected/a877f895-a4cf-404d-87e4-440d4b73a870-kube-api-access-xc8bg\") pod \"node-resolver-ddc5d\" (UID: \"a877f895-a4cf-404d-87e4-440d4b73a870\") " pod="openshift-dns/node-resolver-ddc5d"
Nov 01 05:03:43.291551 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.291485    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n95qs\" (UniqueName: \"kubernetes.io/projected/a9bb6df5-0993-4640-81cc-52426b72fb3e-kube-api-access-n95qs\") pod \"multus-additional-cni-plugins-r4nh2\" (UID: \"a9bb6df5-0993-4640-81cc-52426b72fb3e\") " pod="openshift-multus/multus-additional-cni-plugins-r4nh2"
Nov 01 05:03:43.292480 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.292440    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7npk7\" (UniqueName: \"kubernetes.io/projected/1bd1166d-68e9-4035-b920-e42f5bf05d11-kube-api-access-7npk7\") pod \"console-76cf6f4bb6-dwvbm\" (UID: \"1bd1166d-68e9-4035-b920-e42f5bf05d11\") " pod="openshift-console/console-76cf6f4bb6-dwvbm"
Nov 01 05:03:43.292785 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.292768    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd22d\" (UniqueName: \"kubernetes.io/projected/af0da37a-a117-49a6-a06e-6cab0e56a40c-kube-api-access-hd22d\") pod \"ingress-canary-ljs77\" (UID: \"af0da37a-a117-49a6-a06e-6cab0e56a40c\") " pod="openshift-ingress-canary/ingress-canary-ljs77"
Nov 01 05:03:43.292846 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.292791    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7hhr\" (UniqueName: \"kubernetes.io/projected/80c10aac-5175-46d6-a18b-109169a2461b-kube-api-access-c7hhr\") pod \"thanos-querier-77ddfd4d86-btjkv\" (UID: \"80c10aac-5175-46d6-a18b-109169a2461b\") " pod="openshift-monitoring/thanos-querier-77ddfd4d86-btjkv"
Nov 01 05:03:43.295819 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.295789    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:43.296349 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.296335    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mg57\" (UniqueName: \"kubernetes.io/projected/7cd84f11-44c9-4ec6-83c2-7db17ab9991c-kube-api-access-8mg57\") pod \"migrator-66df5d496b-jn2tf\" (UID: \"7cd84f11-44c9-4ec6-83c2-7db17ab9991c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jn2tf"
Nov 01 05:03:43.296406 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.296391    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5zxr\" (UniqueName: \"kubernetes.io/projected/9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14-kube-api-access-r5zxr\") pod \"node-exporter-vh7rl\" (UID: \"9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14\") " pod="openshift-monitoring/node-exporter-vh7rl"
Nov 01 05:03:43.296437 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.296411    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntxmg\" (UniqueName: \"kubernetes.io/projected/6149f269-49c2-4572-8f6c-fe01e9f0dcbf-kube-api-access-ntxmg\") pod \"iptables-alerter-d8jfj\" (UID: \"6149f269-49c2-4572-8f6c-fe01e9f0dcbf\") " pod="openshift-network-operator/iptables-alerter-d8jfj"
Nov 01 05:03:43.297564 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.297549    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmrx6\" (UniqueName: \"kubernetes.io/projected/7e9cccdd-6c18-4a60-837f-4907411a1c44-kube-api-access-tmrx6\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:03:43.297603 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.297591    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvgxl\" (UniqueName: \"kubernetes.io/projected/805c87a3-63e9-4c13-9c90-c05f73282edc-kube-api-access-hvgxl\") pod \"kube-state-metrics-5dc8d7d89b-5hwhj\" (UID: \"805c87a3-63e9-4c13-9c90-c05f73282edc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5hwhj"
Nov 01 05:03:43.301705 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.301691    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-248gt\" (UniqueName: \"kubernetes.io/projected/6f4aae57-6429-4f9e-aaa5-c700bc26b4f5-kube-api-access-248gt\") pod \"periodic-gathering-t7btw-l6t25\" (UID: \"6f4aae57-6429-4f9e-aaa5-c700bc26b4f5\") " pod="openshift-insights/periodic-gathering-t7btw-l6t25"
Nov 01 05:03:43.303368 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.303353    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rrgq\" (UniqueName: \"kubernetes.io/projected/9752ded2-725f-4035-b374-602e4b4eb576-kube-api-access-8rrgq\") pod \"tuned-zm57v\" (UID: \"9752ded2-725f-4035-b374-602e4b4eb576\") " pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:03:43.307343 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.307323    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wlnj\" (UniqueName: \"kubernetes.io/projected/4321e0ae-e339-40f2-875a-941b1183e766-kube-api-access-2wlnj\") pod \"network-metrics-daemon-267rv\" (UID: \"4321e0ae-e339-40f2-875a-941b1183e766\") " pod="openshift-multus/network-metrics-daemon-267rv"
Nov 01 05:03:43.307399 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.307347    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9njs4\" (UniqueName: \"kubernetes.io/projected/0b0122f6-d51a-4338-a631-056ec9311b69-kube-api-access-9njs4\") pod \"dns-default-g89wb\" (UID: \"0b0122f6-d51a-4338-a631-056ec9311b69\") " pod="openshift-dns/dns-default-g89wb"
Nov 01 05:03:43.307399 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.307331    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgfc6\" (UniqueName: \"kubernetes.io/projected/0e24f687-655b-4edb-8901-35412d3fc79b-kube-api-access-tgfc6\") pod \"aws-ebs-csi-driver-node-6kb4c\" (UID: \"0e24f687-655b-4edb-8901-35412d3fc79b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6kb4c"
Nov 01 05:03:43.310548 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.310530    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zn2l\" (UniqueName: \"kubernetes.io/projected/a8db8643-51bf-4bcf-b26d-bd937a67b5ed-kube-api-access-4zn2l\") pod \"downloads-6dd789c8b7-x7wfp\" (UID: \"a8db8643-51bf-4bcf-b26d-bd937a67b5ed\") " pod="openshift-console/downloads-6dd789c8b7-x7wfp"
Nov 01 05:03:43.310712 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.310692    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55zt7\" (UniqueName: \"kubernetes.io/projected/0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6-kube-api-access-55zt7\") pod \"metrics-server-7b6db58fc-994xl\" (UID: \"0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6\") " pod="openshift-monitoring/metrics-server-7b6db58fc-994xl"
Nov 01 05:03:43.310889 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.310873    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djmg6\" (UniqueName: \"kubernetes.io/projected/46078cac-ae78-4ac7-bda3-b95913a478f8-kube-api-access-djmg6\") pod \"service-ca-5fffc97bdf-pwtc2\" (UID: \"46078cac-ae78-4ac7-bda3-b95913a478f8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-pwtc2"
Nov 01 05:03:43.328870 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.328847    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9752ded2-725f-4035-b374-602e4b4eb576-etc-systemd\") pod \"tuned-zm57v\" (UID: \"9752ded2-725f-4035-b374-602e4b4eb576\") " pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:03:43.328927 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.328876    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9752ded2-725f-4035-b374-602e4b4eb576-run\") pod \"tuned-zm57v\" (UID: \"9752ded2-725f-4035-b374-602e4b4eb576\") " pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:03:43.328927 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.328891    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9752ded2-725f-4035-b374-602e4b4eb576-lib-modules\") pod \"tuned-zm57v\" (UID: \"9752ded2-725f-4035-b374-602e4b4eb576\") " pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:03:43.328927 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.328907    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/972b129f-0166-4705-a71c-09ee0e7a91b9-hostroot\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:03:43.329091 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.328935    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/972b129f-0166-4705-a71c-09ee0e7a91b9-multus-socket-dir-parent\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:03:43.329091 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.328958    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/972b129f-0166-4705-a71c-09ee0e7a91b9-system-cni-dir\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:03:43.329091 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.328966    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9752ded2-725f-4035-b374-602e4b4eb576-run\") pod \"tuned-zm57v\" (UID: \"9752ded2-725f-4035-b374-602e4b4eb576\") " pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:03:43.329091 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.328985    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/972b129f-0166-4705-a71c-09ee0e7a91b9-host-run-netns\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:03:43.329091 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.328996    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9752ded2-725f-4035-b374-602e4b4eb576-etc-systemd\") pod \"tuned-zm57v\" (UID: \"9752ded2-725f-4035-b374-602e4b4eb576\") " pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:03:43.329091 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.329021    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/972b129f-0166-4705-a71c-09ee0e7a91b9-multus-socket-dir-parent\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:03:43.329091 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.329034    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14-root\") pod \"node-exporter-vh7rl\" (UID: \"9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14\") " pod="openshift-monitoring/node-exporter-vh7rl"
Nov 01 05:03:43.329091 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.329008    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14-root\") pod \"node-exporter-vh7rl\" (UID: \"9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14\") " pod="openshift-monitoring/node-exporter-vh7rl"
Nov 01 05:03:43.329091 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.329044    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/972b129f-0166-4705-a71c-09ee0e7a91b9-host-run-netns\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:03:43.329091 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.329043    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9752ded2-725f-4035-b374-602e4b4eb576-lib-modules\") pod \"tuned-zm57v\" (UID: \"9752ded2-725f-4035-b374-602e4b4eb576\") " pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:03:43.329091 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.329044    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/972b129f-0166-4705-a71c-09ee0e7a91b9-system-cni-dir\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:03:43.329091 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.329070    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-run-systemd\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:03:43.329091 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.329089    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-etc-openvswitch\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:03:43.329436 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.329108    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9752ded2-725f-4035-b374-602e4b4eb576-etc-sysconfig\") pod \"tuned-zm57v\" (UID: \"9752ded2-725f-4035-b374-602e4b4eb576\") " pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:03:43.329436 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.329131    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a9bb6df5-0993-4640-81cc-52426b72fb3e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-r4nh2\" (UID: \"a9bb6df5-0993-4640-81cc-52426b72fb3e\") " pod="openshift-multus/multus-additional-cni-plugins-r4nh2"
Nov 01 05:03:43.329436 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.329139    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-run-systemd\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:03:43.329436 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.328996    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/972b129f-0166-4705-a71c-09ee0e7a91b9-hostroot\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:03:43.329436 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.329149    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-etc-openvswitch\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:03:43.329436 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.329148    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0e24f687-655b-4edb-8901-35412d3fc79b-etc-selinux\") pod \"aws-ebs-csi-driver-node-6kb4c\" (UID: \"0e24f687-655b-4edb-8901-35412d3fc79b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6kb4c"
Nov 01 05:03:43.329436 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.329176    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d454e879-03a2-4fb9-8223-79a1aa9d85d2-host\") pod \"node-ca-vrbks\" (UID: \"d454e879-03a2-4fb9-8223-79a1aa9d85d2\") " pod="openshift-image-registry/node-ca-vrbks"
Nov 01 05:03:43.329436 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.329193    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a9bb6df5-0993-4640-81cc-52426b72fb3e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-r4nh2\" (UID: \"a9bb6df5-0993-4640-81cc-52426b72fb3e\") " pod="openshift-multus/multus-additional-cni-plugins-r4nh2"
Nov 01 05:03:43.329436 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.329203    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9752ded2-725f-4035-b374-602e4b4eb576-etc-kubernetes\") pod \"tuned-zm57v\" (UID: \"9752ded2-725f-4035-b374-602e4b4eb576\") " pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:03:43.329436 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.329194    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0e24f687-655b-4edb-8901-35412d3fc79b-etc-selinux\") pod \"aws-ebs-csi-driver-node-6kb4c\" (UID: \"0e24f687-655b-4edb-8901-35412d3fc79b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6kb4c"
Nov 01 05:03:43.329436 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.329216    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d454e879-03a2-4fb9-8223-79a1aa9d85d2-host\") pod \"node-ca-vrbks\" (UID: \"d454e879-03a2-4fb9-8223-79a1aa9d85d2\") " pod="openshift-image-registry/node-ca-vrbks"
Nov 01 05:03:43.329436 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.329220    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9752ded2-725f-4035-b374-602e4b4eb576-etc-sysconfig\") pod \"tuned-zm57v\" (UID: \"9752ded2-725f-4035-b374-602e4b4eb576\") " pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:03:43.329436 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.329222    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/972b129f-0166-4705-a71c-09ee0e7a91b9-host-run-k8s-cni-cncf-io\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:03:43.329436 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.329254    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9752ded2-725f-4035-b374-602e4b4eb576-etc-kubernetes\") pod \"tuned-zm57v\" (UID: \"9752ded2-725f-4035-b374-602e4b4eb576\") " pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:03:43.329436 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.329259    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/972b129f-0166-4705-a71c-09ee0e7a91b9-host-run-k8s-cni-cncf-io\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:03:43.329436 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.329275    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0e24f687-655b-4edb-8901-35412d3fc79b-socket-dir\") pod \"aws-ebs-csi-driver-node-6kb4c\" (UID: \"0e24f687-655b-4edb-8901-35412d3fc79b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6kb4c"
Nov 01 05:03:43.329436 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.329298    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/972b129f-0166-4705-a71c-09ee0e7a91b9-etc-kubernetes\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:03:43.329436 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.329329    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-run-openvswitch\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:03:43.329436 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.329332    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0e24f687-655b-4edb-8901-35412d3fc79b-socket-dir\") pod \"aws-ebs-csi-driver-node-6kb4c\" (UID: \"0e24f687-655b-4edb-8901-35412d3fc79b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6kb4c"
Nov 01 05:03:43.329436 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.329363    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9752ded2-725f-4035-b374-602e4b4eb576-etc-sysctl-conf\") pod \"tuned-zm57v\" (UID: \"9752ded2-725f-4035-b374-602e4b4eb576\") " pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:03:43.329436 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.329366    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-run-openvswitch\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:03:43.329436 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.329382    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/972b129f-0166-4705-a71c-09ee0e7a91b9-etc-kubernetes\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:03:43.329436 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.329390    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/972b129f-0166-4705-a71c-09ee0e7a91b9-host-run-multus-certs\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:03:43.329436 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.329407    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8fba6aca-0a63-4924-89b2-87575d844ce6-crio-socket\") pod \"insights-runtime-extractor-kkng7\" (UID: \"8fba6aca-0a63-4924-89b2-87575d844ce6\") " pod="openshift-insights/insights-runtime-extractor-kkng7"
Nov 01 05:03:43.329436 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.329421    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/972b129f-0166-4705-a71c-09ee0e7a91b9-host-var-lib-cni-multus\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:03:43.329436 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.329437    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-host-kubelet\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:03:43.330495 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.329472    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/972b129f-0166-4705-a71c-09ee0e7a91b9-host-run-multus-certs\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:03:43.330495 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.329479    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-host-run-ovn-kubernetes\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:03:43.330495 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.329509    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/972b129f-0166-4705-a71c-09ee0e7a91b9-host-var-lib-cni-multus\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:03:43.330495 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.329511    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8fba6aca-0a63-4924-89b2-87575d844ce6-crio-socket\") pod \"insights-runtime-extractor-kkng7\" (UID: \"8fba6aca-0a63-4924-89b2-87575d844ce6\") " pod="openshift-insights/insights-runtime-extractor-kkng7"
Nov 01 05:03:43.330495 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.329529    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-host-kubelet\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:03:43.330495 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.329528    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-host-run-ovn-kubernetes\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:03:43.330495 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.329572    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-host-cni-bin\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:03:43.330495 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.329588    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9752ded2-725f-4035-b374-602e4b4eb576-etc-sysctl-conf\") pod \"tuned-zm57v\" (UID: \"9752ded2-725f-4035-b374-602e4b4eb576\") " pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:03:43.330495 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.329598    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-host-run-netns\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:03:43.330495 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.329611    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-host-cni-bin\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:03:43.330495 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.329614    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-node-log\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:03:43.330495 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.329620    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-host-run-netns\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:03:43.330495 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.329634    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-node-log\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:03:43.330495 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.329634    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9752ded2-725f-4035-b374-602e4b4eb576-etc-sysctl-d\") pod \"tuned-zm57v\" (UID: \"9752ded2-725f-4035-b374-602e4b4eb576\") " pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:03:43.330495 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.329666    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9752ded2-725f-4035-b374-602e4b4eb576-sys\") pod \"tuned-zm57v\" (UID: \"9752ded2-725f-4035-b374-602e4b4eb576\") " pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:03:43.330495 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.329684    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a9bb6df5-0993-4640-81cc-52426b72fb3e-system-cni-dir\") pod \"multus-additional-cni-plugins-r4nh2\" (UID: \"a9bb6df5-0993-4640-81cc-52426b72fb3e\") " pod="openshift-multus/multus-additional-cni-plugins-r4nh2"
Nov 01 05:03:43.330495 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.329699    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a877f895-a4cf-404d-87e4-440d4b73a870-hosts-file\") pod \"node-resolver-ddc5d\" (UID: \"a877f895-a4cf-404d-87e4-440d4b73a870\") " pod="openshift-dns/node-resolver-ddc5d"
Nov 01 05:03:43.330495 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.329708    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9752ded2-725f-4035-b374-602e4b4eb576-etc-sysctl-d\") pod \"tuned-zm57v\" (UID: \"9752ded2-725f-4035-b374-602e4b4eb576\") " pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:03:43.330495 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.329713    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/972b129f-0166-4705-a71c-09ee0e7a91b9-multus-conf-dir\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:03:43.330495 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.329734    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a9bb6df5-0993-4640-81cc-52426b72fb3e-system-cni-dir\") pod \"multus-additional-cni-plugins-r4nh2\" (UID: \"a9bb6df5-0993-4640-81cc-52426b72fb3e\") " pod="openshift-multus/multus-additional-cni-plugins-r4nh2"
Nov 01 05:03:43.330495 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.329737    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9752ded2-725f-4035-b374-602e4b4eb576-sys\") pod \"tuned-zm57v\" (UID: \"9752ded2-725f-4035-b374-602e4b4eb576\") " pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:03:43.330495 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.329739    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6149f269-49c2-4572-8f6c-fe01e9f0dcbf-host-slash\") pod \"iptables-alerter-d8jfj\" (UID: \"6149f269-49c2-4572-8f6c-fe01e9f0dcbf\") " pod="openshift-network-operator/iptables-alerter-d8jfj"
Nov 01 05:03:43.330495 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.329762    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a877f895-a4cf-404d-87e4-440d4b73a870-hosts-file\") pod \"node-resolver-ddc5d\" (UID: \"a877f895-a4cf-404d-87e4-440d4b73a870\") " pod="openshift-dns/node-resolver-ddc5d"
Nov 01 05:03:43.330495 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.329771    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6149f269-49c2-4572-8f6c-fe01e9f0dcbf-host-slash\") pod \"iptables-alerter-d8jfj\" (UID: \"6149f269-49c2-4572-8f6c-fe01e9f0dcbf\") " pod="openshift-network-operator/iptables-alerter-d8jfj"
Nov 01 05:03:43.330495 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.329772    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/972b129f-0166-4705-a71c-09ee0e7a91b9-multus-conf-dir\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:03:43.330495 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.329793    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/972b129f-0166-4705-a71c-09ee0e7a91b9-multus-cni-dir\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:03:43.330495 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.329824    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a9bb6df5-0993-4640-81cc-52426b72fb3e-cnibin\") pod \"multus-additional-cni-plugins-r4nh2\" (UID: \"a9bb6df5-0993-4640-81cc-52426b72fb3e\") " pod="openshift-multus/multus-additional-cni-plugins-r4nh2"
Nov 01 05:03:43.330495 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.329850    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0e24f687-655b-4edb-8901-35412d3fc79b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6kb4c\" (UID: \"0e24f687-655b-4edb-8901-35412d3fc79b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6kb4c"
Nov 01 05:03:43.330495 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.329881    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0e24f687-655b-4edb-8901-35412d3fc79b-registration-dir\") pod \"aws-ebs-csi-driver-node-6kb4c\" (UID: \"0e24f687-655b-4edb-8901-35412d3fc79b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6kb4c"
Nov 01 05:03:43.330495 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.329901    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0e24f687-655b-4edb-8901-35412d3fc79b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6kb4c\" (UID: \"0e24f687-655b-4edb-8901-35412d3fc79b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6kb4c"
Nov 01 05:03:43.330495 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.329906    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/972b129f-0166-4705-a71c-09ee0e7a91b9-host-var-lib-kubelet\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:03:43.330495 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.329882    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a9bb6df5-0993-4640-81cc-52426b72fb3e-cnibin\") pod \"multus-additional-cni-plugins-r4nh2\" (UID: \"a9bb6df5-0993-4640-81cc-52426b72fb3e\") " pod="openshift-multus/multus-additional-cni-plugins-r4nh2"
Nov 01 05:03:43.330495 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.329932    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14-node-exporter-wtmp\") pod \"node-exporter-vh7rl\" (UID: \"9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14\") " pod="openshift-monitoring/node-exporter-vh7rl"
Nov 01 05:03:43.330495 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.329951    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/972b129f-0166-4705-a71c-09ee0e7a91b9-host-var-lib-kubelet\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:03:43.330495 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.329857    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/972b129f-0166-4705-a71c-09ee0e7a91b9-multus-cni-dir\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:03:43.330495 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.329963    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0e24f687-655b-4edb-8901-35412d3fc79b-registration-dir\") pod \"aws-ebs-csi-driver-node-6kb4c\" (UID: \"0e24f687-655b-4edb-8901-35412d3fc79b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6kb4c"
Nov 01 05:03:43.330495 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.329955    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-log-socket\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:03:43.330495 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.329990    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9752ded2-725f-4035-b374-602e4b4eb576-host\") pod \"tuned-zm57v\" (UID: \"9752ded2-725f-4035-b374-602e4b4eb576\") " pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:03:43.330495 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.329995    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14-node-exporter-wtmp\") pod \"node-exporter-vh7rl\" (UID: \"9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14\") " pod="openshift-monitoring/node-exporter-vh7rl"
Nov 01 05:03:43.330495 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.329997    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-log-socket\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:03:43.330495 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.330013    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e856df5a-5c5e-4421-ae7d-80a760bc72e2-dbus\") pod \"global-pull-secret-syncer-9pcxf\" (UID: \"e856df5a-5c5e-4421-ae7d-80a760bc72e2\") " pod="kube-system/global-pull-secret-syncer-9pcxf"
Nov 01 05:03:43.330495 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.330032    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9752ded2-725f-4035-b374-602e4b4eb576-etc-modprobe-d\") pod \"tuned-zm57v\" (UID: \"9752ded2-725f-4035-b374-602e4b4eb576\") " pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:03:43.330495 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.330044    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9752ded2-725f-4035-b374-602e4b4eb576-host\") pod \"tuned-zm57v\" (UID: \"9752ded2-725f-4035-b374-602e4b4eb576\") " pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:03:43.330495 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.330054    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e856df5a-5c5e-4421-ae7d-80a760bc72e2-dbus\") pod \"global-pull-secret-syncer-9pcxf\" (UID: \"e856df5a-5c5e-4421-ae7d-80a760bc72e2\") " pod="kube-system/global-pull-secret-syncer-9pcxf"
Nov 01 05:03:43.330495 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.330047    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-host-cni-netd\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:03:43.330495 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.330064    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-host-cni-netd\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:03:43.330495 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.330078    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0e24f687-655b-4edb-8901-35412d3fc79b-device-dir\") pod \"aws-ebs-csi-driver-node-6kb4c\" (UID: \"0e24f687-655b-4edb-8901-35412d3fc79b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6kb4c"
Nov 01 05:03:43.330495 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.330092    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14-sys\") pod \"node-exporter-vh7rl\" (UID: \"9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14\") " pod="openshift-monitoring/node-exporter-vh7rl"
Nov 01 05:03:43.330495 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.330107    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-systemd-units\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:03:43.330495 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.330117    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0e24f687-655b-4edb-8901-35412d3fc79b-device-dir\") pod \"aws-ebs-csi-driver-node-6kb4c\" (UID: \"0e24f687-655b-4edb-8901-35412d3fc79b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6kb4c"
Nov 01 05:03:43.330495 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.330121    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/972b129f-0166-4705-a71c-09ee0e7a91b9-host-var-lib-cni-bin\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:03:43.330495 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.330129    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9752ded2-725f-4035-b374-602e4b4eb576-etc-modprobe-d\") pod \"tuned-zm57v\" (UID: \"9752ded2-725f-4035-b374-602e4b4eb576\") " pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:03:43.330495 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.330138    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/972b129f-0166-4705-a71c-09ee0e7a91b9-host-var-lib-cni-bin\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:03:43.330495 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.330142    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:03:43.330495 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.330161    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-systemd-units\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:03:43.330495 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.330164    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/972b129f-0166-4705-a71c-09ee0e7a91b9-cnibin\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:03:43.330495 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.330173    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14-sys\") pod \"node-exporter-vh7rl\" (UID: \"9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14\") " pod="openshift-monitoring/node-exporter-vh7rl"
Nov 01 05:03:43.330495 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.330184    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:03:43.330495 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.330185    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-host-slash\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:03:43.330495 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.330207    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-host-slash\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:03:43.330495 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.330206    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-var-lib-openvswitch\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:03:43.330495 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.330217    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/972b129f-0166-4705-a71c-09ee0e7a91b9-cnibin\") pod \"multus-5cv94\" (UID: \"972b129f-0166-4705-a71c-09ee0e7a91b9\") " pod="openshift-multus/multus-5cv94"
Nov 01 05:03:43.330495 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.330227    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-run-ovn\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:03:43.330495 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.330231    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-var-lib-openvswitch\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:03:43.330495 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.330245    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9752ded2-725f-4035-b374-602e4b4eb576-var-lib-kubelet\") pod \"tuned-zm57v\" (UID: \"9752ded2-725f-4035-b374-602e4b4eb576\") " pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:03:43.330495 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.330261    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e856df5a-5c5e-4421-ae7d-80a760bc72e2-kubelet-config\") pod \"global-pull-secret-syncer-9pcxf\" (UID: \"e856df5a-5c5e-4421-ae7d-80a760bc72e2\") " pod="kube-system/global-pull-secret-syncer-9pcxf"
Nov 01 05:03:43.330495 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.330276    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0e24f687-655b-4edb-8901-35412d3fc79b-sys-fs\") pod \"aws-ebs-csi-driver-node-6kb4c\" (UID: \"0e24f687-655b-4edb-8901-35412d3fc79b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6kb4c"
Nov 01 05:03:43.330495 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.330295    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9752ded2-725f-4035-b374-602e4b4eb576-var-lib-kubelet\") pod \"tuned-zm57v\" (UID: \"9752ded2-725f-4035-b374-602e4b4eb576\") " pod="openshift-cluster-node-tuning-operator/tuned-zm57v"
Nov 01 05:03:43.330495 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.330324    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e856df5a-5c5e-4421-ae7d-80a760bc72e2-kubelet-config\") pod \"global-pull-secret-syncer-9pcxf\" (UID: \"e856df5a-5c5e-4421-ae7d-80a760bc72e2\") " pod="kube-system/global-pull-secret-syncer-9pcxf"
Nov 01 05:03:43.330495 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.330331    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0e24f687-655b-4edb-8901-35412d3fc79b-sys-fs\") pod \"aws-ebs-csi-driver-node-6kb4c\" (UID: \"0e24f687-655b-4edb-8901-35412d3fc79b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6kb4c"
Nov 01 05:03:43.330495 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.330296    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7e9cccdd-6c18-4a60-837f-4907411a1c44-run-ovn\") pod \"ovnkube-node-2982h\" (UID: \"7e9cccdd-6c18-4a60-837f-4907411a1c44\") " pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:03:43.423293 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.423272    5689 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 05:03:43.925231 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:43.925189    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-645d98cc97-fdph2"]
Nov 01 05:03:44.371192 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:44.371158    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:44.555955 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:44.555862    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:44.759949 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:44.759906    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:44.808918 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:44.807572    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6877799f6d-xrtpt"]
Nov 01 05:03:44.810524 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:44.810499    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6877799f6d-xrtpt"
Nov 01 05:03:44.871815 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:44.871783    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6877799f6d-xrtpt"]
Nov 01 05:03:44.944052 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:44.944017    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e63c480a-630b-4bb6-b4c4-a0f62d781ded-trusted-ca-bundle\") pod \"console-6877799f6d-xrtpt\" (UID: \"e63c480a-630b-4bb6-b4c4-a0f62d781ded\") " pod="openshift-console/console-6877799f6d-xrtpt"
Nov 01 05:03:44.944210 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:44.944072    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e63c480a-630b-4bb6-b4c4-a0f62d781ded-oauth-serving-cert\") pod \"console-6877799f6d-xrtpt\" (UID: \"e63c480a-630b-4bb6-b4c4-a0f62d781ded\") " pod="openshift-console/console-6877799f6d-xrtpt"
Nov 01 05:03:44.944210 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:44.944189    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e63c480a-630b-4bb6-b4c4-a0f62d781ded-console-serving-cert\") pod \"console-6877799f6d-xrtpt\" (UID: \"e63c480a-630b-4bb6-b4c4-a0f62d781ded\") " pod="openshift-console/console-6877799f6d-xrtpt"
Nov 01 05:03:44.944314 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:44.944228    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e63c480a-630b-4bb6-b4c4-a0f62d781ded-service-ca\") pod \"console-6877799f6d-xrtpt\" (UID: \"e63c480a-630b-4bb6-b4c4-a0f62d781ded\") " pod="openshift-console/console-6877799f6d-xrtpt"
Nov 01 05:03:44.944362 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:44.944312    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e63c480a-630b-4bb6-b4c4-a0f62d781ded-console-config\") pod \"console-6877799f6d-xrtpt\" (UID: \"e63c480a-630b-4bb6-b4c4-a0f62d781ded\") " pod="openshift-console/console-6877799f6d-xrtpt"
Nov 01 05:03:44.944411 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:44.944379    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e63c480a-630b-4bb6-b4c4-a0f62d781ded-tmp\") pod \"console-6877799f6d-xrtpt\" (UID: \"e63c480a-630b-4bb6-b4c4-a0f62d781ded\") " pod="openshift-console/console-6877799f6d-xrtpt"
Nov 01 05:03:44.944411 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:44.944402    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlm6w\" (UniqueName: \"kubernetes.io/projected/e63c480a-630b-4bb6-b4c4-a0f62d781ded-kube-api-access-nlm6w\") pod \"console-6877799f6d-xrtpt\" (UID: \"e63c480a-630b-4bb6-b4c4-a0f62d781ded\") " pod="openshift-console/console-6877799f6d-xrtpt"
Nov 01 05:03:44.944536 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:44.944445    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e63c480a-630b-4bb6-b4c4-a0f62d781ded-console-oauth-config\") pod \"console-6877799f6d-xrtpt\" (UID: \"e63c480a-630b-4bb6-b4c4-a0f62d781ded\") " pod="openshift-console/console-6877799f6d-xrtpt"
Nov 01 05:03:45.045114 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:45.045077    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e63c480a-630b-4bb6-b4c4-a0f62d781ded-console-config\") pod \"console-6877799f6d-xrtpt\" (UID: \"e63c480a-630b-4bb6-b4c4-a0f62d781ded\") " pod="openshift-console/console-6877799f6d-xrtpt"
Nov 01 05:03:45.045281 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:45.045131    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e63c480a-630b-4bb6-b4c4-a0f62d781ded-tmp\") pod \"console-6877799f6d-xrtpt\" (UID: \"e63c480a-630b-4bb6-b4c4-a0f62d781ded\") " pod="openshift-console/console-6877799f6d-xrtpt"
Nov 01 05:03:45.045281 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:45.045156    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlm6w\" (UniqueName: \"kubernetes.io/projected/e63c480a-630b-4bb6-b4c4-a0f62d781ded-kube-api-access-nlm6w\") pod \"console-6877799f6d-xrtpt\" (UID: \"e63c480a-630b-4bb6-b4c4-a0f62d781ded\") " pod="openshift-console/console-6877799f6d-xrtpt"
Nov 01 05:03:45.045281 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:45.045211    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e63c480a-630b-4bb6-b4c4-a0f62d781ded-console-oauth-config\") pod \"console-6877799f6d-xrtpt\" (UID: \"e63c480a-630b-4bb6-b4c4-a0f62d781ded\") " pod="openshift-console/console-6877799f6d-xrtpt"
Nov 01 05:03:45.045281 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:45.045264    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e63c480a-630b-4bb6-b4c4-a0f62d781ded-trusted-ca-bundle\") pod \"console-6877799f6d-xrtpt\" (UID: \"e63c480a-630b-4bb6-b4c4-a0f62d781ded\") " pod="openshift-console/console-6877799f6d-xrtpt"
Nov 01 05:03:45.045563 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:45.045307    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e63c480a-630b-4bb6-b4c4-a0f62d781ded-oauth-serving-cert\") pod \"console-6877799f6d-xrtpt\" (UID: \"e63c480a-630b-4bb6-b4c4-a0f62d781ded\") " pod="openshift-console/console-6877799f6d-xrtpt"
Nov 01 05:03:45.045563 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:45.045352    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e63c480a-630b-4bb6-b4c4-a0f62d781ded-console-serving-cert\") pod \"console-6877799f6d-xrtpt\" (UID: \"e63c480a-630b-4bb6-b4c4-a0f62d781ded\") " pod="openshift-console/console-6877799f6d-xrtpt"
Nov 01 05:03:45.045563 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:45.045373    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e63c480a-630b-4bb6-b4c4-a0f62d781ded-service-ca\") pod \"console-6877799f6d-xrtpt\" (UID: \"e63c480a-630b-4bb6-b4c4-a0f62d781ded\") " pod="openshift-console/console-6877799f6d-xrtpt"
Nov 01 05:03:45.045740 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:45.045712    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e63c480a-630b-4bb6-b4c4-a0f62d781ded-tmp\") pod \"console-6877799f6d-xrtpt\" (UID: \"e63c480a-630b-4bb6-b4c4-a0f62d781ded\") " pod="openshift-console/console-6877799f6d-xrtpt"
Nov 01 05:03:45.045826 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:45.045797    5689 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 01 05:03:45.045986 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:45.045962    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e63c480a-630b-4bb6-b4c4-a0f62d781ded-console-config\") pod \"console-6877799f6d-xrtpt\" (UID: \"e63c480a-630b-4bb6-b4c4-a0f62d781ded\") " pod="openshift-console/console-6877799f6d-xrtpt"
Nov 01 05:03:45.046339 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:45.046311    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e63c480a-630b-4bb6-b4c4-a0f62d781ded-trusted-ca-bundle\") pod \"console-6877799f6d-xrtpt\" (UID: \"e63c480a-630b-4bb6-b4c4-a0f62d781ded\") " pod="openshift-console/console-6877799f6d-xrtpt"
Nov 01 05:03:45.046911 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:45.046882    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e63c480a-630b-4bb6-b4c4-a0f62d781ded-oauth-serving-cert\") pod \"console-6877799f6d-xrtpt\" (UID: \"e63c480a-630b-4bb6-b4c4-a0f62d781ded\") " pod="openshift-console/console-6877799f6d-xrtpt"
Nov 01 05:03:45.047169 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:45.046947    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e63c480a-630b-4bb6-b4c4-a0f62d781ded-service-ca\") pod \"console-6877799f6d-xrtpt\" (UID: \"e63c480a-630b-4bb6-b4c4-a0f62d781ded\") " pod="openshift-console/console-6877799f6d-xrtpt"
Nov 01 05:03:45.048675 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:45.048647    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e63c480a-630b-4bb6-b4c4-a0f62d781ded-console-oauth-config\") pod \"console-6877799f6d-xrtpt\" (UID: \"e63c480a-630b-4bb6-b4c4-a0f62d781ded\") " pod="openshift-console/console-6877799f6d-xrtpt"
Nov 01 05:03:45.049135 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:45.049116    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e63c480a-630b-4bb6-b4c4-a0f62d781ded-console-serving-cert\") pod \"console-6877799f6d-xrtpt\" (UID: \"e63c480a-630b-4bb6-b4c4-a0f62d781ded\") " pod="openshift-console/console-6877799f6d-xrtpt"
Nov 01 05:03:45.106582 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:45.106505    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlm6w\" (UniqueName: \"kubernetes.io/projected/e63c480a-630b-4bb6-b4c4-a0f62d781ded-kube-api-access-nlm6w\") pod \"console-6877799f6d-xrtpt\" (UID: \"e63c480a-630b-4bb6-b4c4-a0f62d781ded\") " pod="openshift-console/console-6877799f6d-xrtpt"
Nov 01 05:03:45.123254 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:45.123224    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6877799f6d-xrtpt"
Nov 01 05:03:45.142055 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:45.140496    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-x7wfp" event={"ID":"a8db8643-51bf-4bcf-b26d-bd937a67b5ed","Type":"ContainerStarted","Data":"dd29c3bb1627fc0ba067c00d32e802407954cabbe61413e740cd66aada7ed1a0"}
Nov 01 05:03:45.153677 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:45.152319    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-77ddfd4d86-btjkv"
Nov 01 05:03:45.153814 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:45.153739    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-fbbdp"
Nov 01 05:03:45.158389 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:45.156096    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-77ddfd4d86-btjkv"
Nov 01 05:03:45.266693 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:45.266668    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:03:45.310937 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:45.310904    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7b6db58fc-994xl"
Nov 01 05:03:45.518702 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:45.518663    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6877799f6d-xrtpt"]
Nov 01 05:03:45.523926 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:03:45.523881    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode63c480a_630b_4bb6_b4c4_a0f62d781ded.slice/crio-f82252073384d9ab4935bbd3c186b32e634315b8d1ac5bc19c7ddcf6c21a4adc WatchSource:0}: Error finding container f82252073384d9ab4935bbd3c186b32e634315b8d1ac5bc19c7ddcf6c21a4adc: Status 404 returned error can't find the container with id f82252073384d9ab4935bbd3c186b32e634315b8d1ac5bc19c7ddcf6c21a4adc
Nov 01 05:03:45.556050 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:45.556024    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-g89wb"
Nov 01 05:03:45.604057 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:45.604028    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-645d98cc97-fdph2"
Nov 01 05:03:45.786284 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:45.786196    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:45.971024 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:45.970979    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:46.146787 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:46.146704    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6877799f6d-xrtpt" event={"ID":"e63c480a-630b-4bb6-b4c4-a0f62d781ded","Type":"ContainerStarted","Data":"c8f75368b6780c1a35df5bdf008ff4dcb9da2b75a2b6aa1c4e425ea36e0bc7cb"}
Nov 01 05:03:46.146787 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:46.146768    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6877799f6d-xrtpt" event={"ID":"e63c480a-630b-4bb6-b4c4-a0f62d781ded","Type":"ContainerStarted","Data":"f82252073384d9ab4935bbd3c186b32e634315b8d1ac5bc19c7ddcf6c21a4adc"}
Nov 01 05:03:46.265808 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:46.264793    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6877799f6d-xrtpt" podStartSLOduration=2.264773742 podStartE2EDuration="2.264773742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:03:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:03:46.263627222 +0000 UTC m=+4.334424154" watchObservedRunningTime="2026-11-01 05:03:46.264773742 +0000 UTC m=+4.335570515"
Nov 01 05:03:47.977923 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:47.977884    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:03:47.978327 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:47.978153    5689 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 01 05:03:49.115668 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:49.115632    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-76cf6f4bb6-dwvbm"
Nov 01 05:03:49.120119 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:49.120090    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-76cf6f4bb6-dwvbm"
Nov 01 05:03:49.203784 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:49.203743    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6877799f6d-xrtpt"
Nov 01 05:03:49.209431 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:49.209407    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6877799f6d-xrtpt"
Nov 01 05:03:50.755501 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:50.755392    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-x7wfp"
Nov 01 05:03:50.756065 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:50.755602    5689 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 01 05:03:50.758389 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:50.758359    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-x7wfp"
Nov 01 05:03:51.060536 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:51.060448    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-sr4n8"
Nov 01 05:03:51.976953 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:51.976911    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-76cf6f4bb6-dwvbm"
Nov 01 05:03:53.388775 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:53.388741    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:53.595600 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:53.595568    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:53.626234 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:53.626206    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6877799f6d-xrtpt"
Nov 01 05:03:53.626391 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:53.626377    5689 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 01 05:03:53.630277 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:53.630255    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6877799f6d-xrtpt"
Nov 01 05:03:53.791200 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:53.791168    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:53.798204 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:53.798179    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-76cf6f4bb6-dwvbm"]
Nov 01 05:03:58.066631 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:58.066597    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:03:58.252001 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:03:58.251966    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:04:00.071704 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:04:00.071673    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:04:02.082608 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:02.082579    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7b6db58fc-994xl"
Nov 01 05:04:05.316060 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:05.316030    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7b6db58fc-994xl"
Nov 01 05:04:05.319797 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:05.319774    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7b6db58fc-994xl"
Nov 01 05:04:09.153403 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:09.153361    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-645d98cc97-fdph2" podUID="ca43c15d-cefb-4e78-93aa-a98573da4feb" containerName="console" containerID="cri-o://ab788a84b045eb570edb9df7c5f6bebf57601753563f826de606de4849b6d5f1" gracePeriod=15
Nov 01 05:04:09.435140 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:09.435080    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-645d98cc97-fdph2"
Nov 01 05:04:09.547621 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:09.547588    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca43c15d-cefb-4e78-93aa-a98573da4feb-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ca43c15d-cefb-4e78-93aa-a98573da4feb-console-oauth-config\") pod \"ca43c15d-cefb-4e78-93aa-a98573da4feb\" (UID: \"ca43c15d-cefb-4e78-93aa-a98573da4feb\") "
Nov 01 05:04:09.547791 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:09.547646    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca43c15d-cefb-4e78-93aa-a98573da4feb-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca43c15d-cefb-4e78-93aa-a98573da4feb-service-ca\") pod \"ca43c15d-cefb-4e78-93aa-a98573da4feb\" (UID: \"ca43c15d-cefb-4e78-93aa-a98573da4feb\") "
Nov 01 05:04:09.547791 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:09.547697    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca43c15d-cefb-4e78-93aa-a98573da4feb-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ca43c15d-cefb-4e78-93aa-a98573da4feb-console-serving-cert\") pod \"ca43c15d-cefb-4e78-93aa-a98573da4feb\" (UID: \"ca43c15d-cefb-4e78-93aa-a98573da4feb\") "
Nov 01 05:04:09.547791 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:09.547724    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca43c15d-cefb-4e78-93aa-a98573da4feb-console-config\" (UniqueName: \"kubernetes.io/configmap/ca43c15d-cefb-4e78-93aa-a98573da4feb-console-config\") pod \"ca43c15d-cefb-4e78-93aa-a98573da4feb\" (UID: \"ca43c15d-cefb-4e78-93aa-a98573da4feb\") "
Nov 01 05:04:09.547791 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:09.547754    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca43c15d-cefb-4e78-93aa-a98573da4feb-kube-api-access-wgwkh\" (UniqueName: \"kubernetes.io/projected/ca43c15d-cefb-4e78-93aa-a98573da4feb-kube-api-access-wgwkh\") pod \"ca43c15d-cefb-4e78-93aa-a98573da4feb\" (UID: \"ca43c15d-cefb-4e78-93aa-a98573da4feb\") "
Nov 01 05:04:09.548046 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:09.547804    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca43c15d-cefb-4e78-93aa-a98573da4feb-tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca43c15d-cefb-4e78-93aa-a98573da4feb-tmp\") pod \"ca43c15d-cefb-4e78-93aa-a98573da4feb\" (UID: \"ca43c15d-cefb-4e78-93aa-a98573da4feb\") "
Nov 01 05:04:09.548046 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:09.547969    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca43c15d-cefb-4e78-93aa-a98573da4feb-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ca43c15d-cefb-4e78-93aa-a98573da4feb-oauth-serving-cert\") pod \"ca43c15d-cefb-4e78-93aa-a98573da4feb\" (UID: \"ca43c15d-cefb-4e78-93aa-a98573da4feb\") "
Nov 01 05:04:09.548150 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:09.548076    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca43c15d-cefb-4e78-93aa-a98573da4feb-tmp" pod "ca43c15d-cefb-4e78-93aa-a98573da4feb" (UID: "ca43c15d-cefb-4e78-93aa-a98573da4feb"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:04:09.548150 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:09.548097    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca43c15d-cefb-4e78-93aa-a98573da4feb-service-ca" pod "ca43c15d-cefb-4e78-93aa-a98573da4feb" (UID: "ca43c15d-cefb-4e78-93aa-a98573da4feb"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:04:09.548280 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:09.548259    5689 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ca43c15d-cefb-4e78-93aa-a98573da4feb-service-ca\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:04:09.548517 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:09.548287    5689 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca43c15d-cefb-4e78-93aa-a98573da4feb-tmp\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:04:09.548517 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:09.548301    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca43c15d-cefb-4e78-93aa-a98573da4feb-console-config" pod "ca43c15d-cefb-4e78-93aa-a98573da4feb" (UID: "ca43c15d-cefb-4e78-93aa-a98573da4feb"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:04:09.548517 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:09.548369    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca43c15d-cefb-4e78-93aa-a98573da4feb-oauth-serving-cert" pod "ca43c15d-cefb-4e78-93aa-a98573da4feb" (UID: "ca43c15d-cefb-4e78-93aa-a98573da4feb"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:04:09.550139 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:09.550111    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca43c15d-cefb-4e78-93aa-a98573da4feb-console-oauth-config" pod "ca43c15d-cefb-4e78-93aa-a98573da4feb" (UID: "ca43c15d-cefb-4e78-93aa-a98573da4feb"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 05:04:09.550226 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:09.550145    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca43c15d-cefb-4e78-93aa-a98573da4feb-kube-api-access-wgwkh" pod "ca43c15d-cefb-4e78-93aa-a98573da4feb" (UID: "ca43c15d-cefb-4e78-93aa-a98573da4feb"). InnerVolumeSpecName "kube-api-access-wgwkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:04:09.550226 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:09.550179    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca43c15d-cefb-4e78-93aa-a98573da4feb-console-serving-cert" pod "ca43c15d-cefb-4e78-93aa-a98573da4feb" (UID: "ca43c15d-cefb-4e78-93aa-a98573da4feb"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 05:04:09.649586 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:09.649562    5689 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ca43c15d-cefb-4e78-93aa-a98573da4feb-oauth-serving-cert\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:04:09.649586 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:09.649584    5689 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ca43c15d-cefb-4e78-93aa-a98573da4feb-console-oauth-config\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:04:09.649724 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:09.649594    5689 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ca43c15d-cefb-4e78-93aa-a98573da4feb-console-serving-cert\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:04:09.649724 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:09.649602    5689 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ca43c15d-cefb-4e78-93aa-a98573da4feb-console-config\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:04:09.649724 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:09.649611    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgwkh\" (UniqueName: \"kubernetes.io/projected/ca43c15d-cefb-4e78-93aa-a98573da4feb-kube-api-access-wgwkh\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:04:10.240310 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:10.240276    5689 generic.go:356] "Generic (PLEG): container finished" podID="ca43c15d-cefb-4e78-93aa-a98573da4feb" containerID="ab788a84b045eb570edb9df7c5f6bebf57601753563f826de606de4849b6d5f1" exitCode=0
Nov 01 05:04:10.240715 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:10.240321    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-645d98cc97-fdph2"
Nov 01 05:04:10.240715 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:10.240343    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-645d98cc97-fdph2" event={"ID":"ca43c15d-cefb-4e78-93aa-a98573da4feb","Type":"ContainerDied","Data":"ab788a84b045eb570edb9df7c5f6bebf57601753563f826de606de4849b6d5f1"}
Nov 01 05:04:10.240715 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:10.240387    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-645d98cc97-fdph2" event={"ID":"ca43c15d-cefb-4e78-93aa-a98573da4feb","Type":"ContainerDied","Data":"78cfa2ceed0dfad5311f2c8230151b15c354cce14a4b7b45aa09dac43e4efa53"}
Nov 01 05:04:10.240715 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:10.240416    5689 scope.go:122] "RemoveContainer" containerID="ab788a84b045eb570edb9df7c5f6bebf57601753563f826de606de4849b6d5f1"
Nov 01 05:04:10.255774 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:10.255751    5689 scope.go:122] "RemoveContainer" containerID="ab788a84b045eb570edb9df7c5f6bebf57601753563f826de606de4849b6d5f1"
Nov 01 05:04:10.256025 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:04:10.256003    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab788a84b045eb570edb9df7c5f6bebf57601753563f826de606de4849b6d5f1\": container with ID starting with ab788a84b045eb570edb9df7c5f6bebf57601753563f826de606de4849b6d5f1 not found: ID does not exist" containerID="ab788a84b045eb570edb9df7c5f6bebf57601753563f826de606de4849b6d5f1"
Nov 01 05:04:10.256106 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:10.256036    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab788a84b045eb570edb9df7c5f6bebf57601753563f826de606de4849b6d5f1"} err="failed to get container status \"ab788a84b045eb570edb9df7c5f6bebf57601753563f826de606de4849b6d5f1\": rpc error: code = NotFound desc = could not find container \"ab788a84b045eb570edb9df7c5f6bebf57601753563f826de606de4849b6d5f1\": container with ID starting with ab788a84b045eb570edb9df7c5f6bebf57601753563f826de606de4849b6d5f1 not found: ID does not exist"
Nov 01 05:04:10.406324 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:10.406297    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-645d98cc97-fdph2"]
Nov 01 05:04:10.454908 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:10.454881    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-645d98cc97-fdph2"]
Nov 01 05:04:11.985401 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:11.985377    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2982h"
Nov 01 05:04:12.067395 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:12.067365    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca43c15d-cefb-4e78-93aa-a98573da4feb" path="/var/lib/kubelet/pods/ca43c15d-cefb-4e78-93aa-a98573da4feb/volumes"
Nov 01 05:04:12.440519 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:12.440490    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-hzr4h"
Nov 01 05:04:14.249788 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:14.249748    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:04:14.425933 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:14.425902    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:04:18.817760 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:18.817693    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-76cf6f4bb6-dwvbm" podUID="1bd1166d-68e9-4035-b920-e42f5bf05d11" containerName="console" containerID="cri-o://c7ab7d992cbad4554a3f823f7d45f7129516fc079ac30429bb2b139a4c635ffc" gracePeriod=15
Nov 01 05:04:19.075660 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:19.075606    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76cf6f4bb6-dwvbm"
Nov 01 05:04:19.214387 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:19.214350    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1bd1166d-68e9-4035-b920-e42f5bf05d11-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1bd1166d-68e9-4035-b920-e42f5bf05d11-console-serving-cert\") pod \"1bd1166d-68e9-4035-b920-e42f5bf05d11\" (UID: \"1bd1166d-68e9-4035-b920-e42f5bf05d11\") "
Nov 01 05:04:19.214561 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:19.214406    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bd1166d-68e9-4035-b920-e42f5bf05d11-kube-api-access-7npk7\" (UniqueName: \"kubernetes.io/projected/1bd1166d-68e9-4035-b920-e42f5bf05d11-kube-api-access-7npk7\") pod \"1bd1166d-68e9-4035-b920-e42f5bf05d11\" (UID: \"1bd1166d-68e9-4035-b920-e42f5bf05d11\") "
Nov 01 05:04:19.214561 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:19.214488    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1bd1166d-68e9-4035-b920-e42f5bf05d11-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1bd1166d-68e9-4035-b920-e42f5bf05d11-console-oauth-config\") pod \"1bd1166d-68e9-4035-b920-e42f5bf05d11\" (UID: \"1bd1166d-68e9-4035-b920-e42f5bf05d11\") "
Nov 01 05:04:19.214676 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:19.214561    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bd1166d-68e9-4035-b920-e42f5bf05d11-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1bd1166d-68e9-4035-b920-e42f5bf05d11-oauth-serving-cert\") pod \"1bd1166d-68e9-4035-b920-e42f5bf05d11\" (UID: \"1bd1166d-68e9-4035-b920-e42f5bf05d11\") "
Nov 01 05:04:19.214676 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:19.214593    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bd1166d-68e9-4035-b920-e42f5bf05d11-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bd1166d-68e9-4035-b920-e42f5bf05d11-service-ca\") pod \"1bd1166d-68e9-4035-b920-e42f5bf05d11\" (UID: \"1bd1166d-68e9-4035-b920-e42f5bf05d11\") "
Nov 01 05:04:19.214676 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:19.214629    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bd1166d-68e9-4035-b920-e42f5bf05d11-console-config\" (UniqueName: \"kubernetes.io/configmap/1bd1166d-68e9-4035-b920-e42f5bf05d11-console-config\") pod \"1bd1166d-68e9-4035-b920-e42f5bf05d11\" (UID: \"1bd1166d-68e9-4035-b920-e42f5bf05d11\") "
Nov 01 05:04:19.214676 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:19.214668    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bd1166d-68e9-4035-b920-e42f5bf05d11-tmp\" (UniqueName: \"kubernetes.io/empty-dir/1bd1166d-68e9-4035-b920-e42f5bf05d11-tmp\") pod \"1bd1166d-68e9-4035-b920-e42f5bf05d11\" (UID: \"1bd1166d-68e9-4035-b920-e42f5bf05d11\") "
Nov 01 05:04:19.214866 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:19.214703    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bd1166d-68e9-4035-b920-e42f5bf05d11-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1bd1166d-68e9-4035-b920-e42f5bf05d11-trusted-ca-bundle\") pod \"1bd1166d-68e9-4035-b920-e42f5bf05d11\" (UID: \"1bd1166d-68e9-4035-b920-e42f5bf05d11\") "
Nov 01 05:04:19.214998 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:19.214946    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bd1166d-68e9-4035-b920-e42f5bf05d11-oauth-serving-cert" pod "1bd1166d-68e9-4035-b920-e42f5bf05d11" (UID: "1bd1166d-68e9-4035-b920-e42f5bf05d11"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:04:19.214998 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:19.214963    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bd1166d-68e9-4035-b920-e42f5bf05d11-service-ca" pod "1bd1166d-68e9-4035-b920-e42f5bf05d11" (UID: "1bd1166d-68e9-4035-b920-e42f5bf05d11"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:04:19.215484 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:19.215318    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bd1166d-68e9-4035-b920-e42f5bf05d11-console-config" pod "1bd1166d-68e9-4035-b920-e42f5bf05d11" (UID: "1bd1166d-68e9-4035-b920-e42f5bf05d11"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:04:19.218779 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:19.218739    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bd1166d-68e9-4035-b920-e42f5bf05d11-tmp" pod "1bd1166d-68e9-4035-b920-e42f5bf05d11" (UID: "1bd1166d-68e9-4035-b920-e42f5bf05d11"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:04:19.218893 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:19.218874    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bd1166d-68e9-4035-b920-e42f5bf05d11-trusted-ca-bundle" pod "1bd1166d-68e9-4035-b920-e42f5bf05d11" (UID: "1bd1166d-68e9-4035-b920-e42f5bf05d11"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:04:19.218955 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:19.218896    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bd1166d-68e9-4035-b920-e42f5bf05d11-kube-api-access-7npk7" pod "1bd1166d-68e9-4035-b920-e42f5bf05d11" (UID: "1bd1166d-68e9-4035-b920-e42f5bf05d11"). InnerVolumeSpecName "kube-api-access-7npk7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:04:19.218955 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:19.218904    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bd1166d-68e9-4035-b920-e42f5bf05d11-console-oauth-config" pod "1bd1166d-68e9-4035-b920-e42f5bf05d11" (UID: "1bd1166d-68e9-4035-b920-e42f5bf05d11"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 05:04:19.219061 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:19.218988    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bd1166d-68e9-4035-b920-e42f5bf05d11-console-serving-cert" pod "1bd1166d-68e9-4035-b920-e42f5bf05d11" (UID: "1bd1166d-68e9-4035-b920-e42f5bf05d11"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 05:04:19.270609 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:19.270582    5689 generic.go:356] "Generic (PLEG): container finished" podID="1bd1166d-68e9-4035-b920-e42f5bf05d11" containerID="c7ab7d992cbad4554a3f823f7d45f7129516fc079ac30429bb2b139a4c635ffc" exitCode=0
Nov 01 05:04:19.270702 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:19.270634    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76cf6f4bb6-dwvbm"
Nov 01 05:04:19.270702 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:19.270664    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76cf6f4bb6-dwvbm" event={"ID":"1bd1166d-68e9-4035-b920-e42f5bf05d11","Type":"ContainerDied","Data":"c7ab7d992cbad4554a3f823f7d45f7129516fc079ac30429bb2b139a4c635ffc"}
Nov 01 05:04:19.270769 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:19.270712    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76cf6f4bb6-dwvbm" event={"ID":"1bd1166d-68e9-4035-b920-e42f5bf05d11","Type":"ContainerDied","Data":"94b090bb2f81614395a6c92d20a127033288b6317a3746960c12ca9c06f937c8"}
Nov 01 05:04:19.270769 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:19.270731    5689 scope.go:122] "RemoveContainer" containerID="c7ab7d992cbad4554a3f823f7d45f7129516fc079ac30429bb2b139a4c635ffc"
Nov 01 05:04:19.278438 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:19.278418    5689 scope.go:122] "RemoveContainer" containerID="c7ab7d992cbad4554a3f823f7d45f7129516fc079ac30429bb2b139a4c635ffc"
Nov 01 05:04:19.278706 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:04:19.278687    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7ab7d992cbad4554a3f823f7d45f7129516fc079ac30429bb2b139a4c635ffc\": container with ID starting with c7ab7d992cbad4554a3f823f7d45f7129516fc079ac30429bb2b139a4c635ffc not found: ID does not exist" containerID="c7ab7d992cbad4554a3f823f7d45f7129516fc079ac30429bb2b139a4c635ffc"
Nov 01 05:04:19.278757 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:19.278715    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7ab7d992cbad4554a3f823f7d45f7129516fc079ac30429bb2b139a4c635ffc"} err="failed to get container status \"c7ab7d992cbad4554a3f823f7d45f7129516fc079ac30429bb2b139a4c635ffc\": rpc error: code = NotFound desc = could not find container \"c7ab7d992cbad4554a3f823f7d45f7129516fc079ac30429bb2b139a4c635ffc\": container with ID starting with c7ab7d992cbad4554a3f823f7d45f7129516fc079ac30429bb2b139a4c635ffc not found: ID does not exist"
Nov 01 05:04:19.316056 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:19.316039    5689 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1bd1166d-68e9-4035-b920-e42f5bf05d11-service-ca\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:04:19.316134 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:19.316057    5689 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1bd1166d-68e9-4035-b920-e42f5bf05d11-console-config\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:04:19.316134 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:19.316068    5689 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1bd1166d-68e9-4035-b920-e42f5bf05d11-tmp\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:04:19.316134 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:19.316076    5689 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1bd1166d-68e9-4035-b920-e42f5bf05d11-trusted-ca-bundle\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:04:19.316134 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:19.316085    5689 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1bd1166d-68e9-4035-b920-e42f5bf05d11-console-serving-cert\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:04:19.316134 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:19.316094    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7npk7\" (UniqueName: \"kubernetes.io/projected/1bd1166d-68e9-4035-b920-e42f5bf05d11-kube-api-access-7npk7\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:04:19.316134 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:19.316103    5689 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1bd1166d-68e9-4035-b920-e42f5bf05d11-console-oauth-config\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:04:19.316134 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:19.316112    5689 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1bd1166d-68e9-4035-b920-e42f5bf05d11-oauth-serving-cert\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:04:19.431859 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:19.431830    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-76cf6f4bb6-dwvbm"]
Nov 01 05:04:19.435586 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:19.435567    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-76cf6f4bb6-dwvbm"]
Nov 01 05:04:20.065345 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:20.065307    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bd1166d-68e9-4035-b920-e42f5bf05d11" path="/var/lib/kubelet/pods/1bd1166d-68e9-4035-b920-e42f5bf05d11/volumes"
Nov 01 05:04:23.729731 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:23.729691    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:04:23.906315 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:23.906279    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:04:25.577115 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:25.577074    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-t7btw-l6t25"]
Nov 01 05:04:25.577742 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:25.577698    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-t7btw-l6t25" podUID="6f4aae57-6429-4f9e-aaa5-c700bc26b4f5" containerName="insights-gathering" containerID="cri-o://926cf2e9417e15c25ee90ffb91fa457785273e7901c6f240cbc6763bc47826a8" gracePeriod=30
Nov 01 05:04:25.859181 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:25.859159    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-t7btw-l6t25"
Nov 01 05:04:25.960192 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:25.960168    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f4aae57-6429-4f9e-aaa5-c700bc26b4f5-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/6f4aae57-6429-4f9e-aaa5-c700bc26b4f5-archives-path\") pod \"6f4aae57-6429-4f9e-aaa5-c700bc26b4f5\" (UID: \"6f4aae57-6429-4f9e-aaa5-c700bc26b4f5\") "
Nov 01 05:04:25.960292 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:25.960203    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f4aae57-6429-4f9e-aaa5-c700bc26b4f5-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f4aae57-6429-4f9e-aaa5-c700bc26b4f5-trusted-ca-bundle\") pod \"6f4aae57-6429-4f9e-aaa5-c700bc26b4f5\" (UID: \"6f4aae57-6429-4f9e-aaa5-c700bc26b4f5\") "
Nov 01 05:04:25.960292 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:25.960261    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f4aae57-6429-4f9e-aaa5-c700bc26b4f5-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f4aae57-6429-4f9e-aaa5-c700bc26b4f5-service-ca-bundle\") pod \"6f4aae57-6429-4f9e-aaa5-c700bc26b4f5\" (UID: \"6f4aae57-6429-4f9e-aaa5-c700bc26b4f5\") "
Nov 01 05:04:25.960366 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:25.960355    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f4aae57-6429-4f9e-aaa5-c700bc26b4f5-kube-api-access-248gt\" (UniqueName: \"kubernetes.io/projected/6f4aae57-6429-4f9e-aaa5-c700bc26b4f5-kube-api-access-248gt\") pod \"6f4aae57-6429-4f9e-aaa5-c700bc26b4f5\" (UID: \"6f4aae57-6429-4f9e-aaa5-c700bc26b4f5\") "
Nov 01 05:04:25.960534 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:25.960511    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f4aae57-6429-4f9e-aaa5-c700bc26b4f5-archives-path" pod "6f4aae57-6429-4f9e-aaa5-c700bc26b4f5" (UID: "6f4aae57-6429-4f9e-aaa5-c700bc26b4f5"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:04:25.960626 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:25.960599    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f4aae57-6429-4f9e-aaa5-c700bc26b4f5-trusted-ca-bundle" pod "6f4aae57-6429-4f9e-aaa5-c700bc26b4f5" (UID: "6f4aae57-6429-4f9e-aaa5-c700bc26b4f5"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:04:25.960694 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:25.960625    5689 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/6f4aae57-6429-4f9e-aaa5-c700bc26b4f5-archives-path\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:04:25.960694 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:25.960681    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f4aae57-6429-4f9e-aaa5-c700bc26b4f5-service-ca-bundle" pod "6f4aae57-6429-4f9e-aaa5-c700bc26b4f5" (UID: "6f4aae57-6429-4f9e-aaa5-c700bc26b4f5"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:04:25.962663 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:25.962641    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f4aae57-6429-4f9e-aaa5-c700bc26b4f5-kube-api-access-248gt" pod "6f4aae57-6429-4f9e-aaa5-c700bc26b4f5" (UID: "6f4aae57-6429-4f9e-aaa5-c700bc26b4f5"). InnerVolumeSpecName "kube-api-access-248gt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:04:26.061213 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:26.061187    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-248gt\" (UniqueName: \"kubernetes.io/projected/6f4aae57-6429-4f9e-aaa5-c700bc26b4f5-kube-api-access-248gt\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:04:26.061213 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:26.061208    5689 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f4aae57-6429-4f9e-aaa5-c700bc26b4f5-trusted-ca-bundle\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:04:26.061213 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:26.061218    5689 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f4aae57-6429-4f9e-aaa5-c700bc26b4f5-service-ca-bundle\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:04:26.295788 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:26.295752    5689 generic.go:356] "Generic (PLEG): container finished" podID="6f4aae57-6429-4f9e-aaa5-c700bc26b4f5" containerID="926cf2e9417e15c25ee90ffb91fa457785273e7901c6f240cbc6763bc47826a8" exitCode=2
Nov 01 05:04:26.295925 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:26.295805    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-t7btw-l6t25"
Nov 01 05:04:26.295925 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:26.295833    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-t7btw-l6t25" event={"ID":"6f4aae57-6429-4f9e-aaa5-c700bc26b4f5","Type":"ContainerDied","Data":"926cf2e9417e15c25ee90ffb91fa457785273e7901c6f240cbc6763bc47826a8"}
Nov 01 05:04:26.295925 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:26.295885    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-t7btw-l6t25" event={"ID":"6f4aae57-6429-4f9e-aaa5-c700bc26b4f5","Type":"ContainerDied","Data":"d1f3f2d5ced62268ee33102b7bc84b719b90c807b6ef49c26c5eec98182f2d13"}
Nov 01 05:04:26.295925 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:26.295904    5689 scope.go:122] "RemoveContainer" containerID="926cf2e9417e15c25ee90ffb91fa457785273e7901c6f240cbc6763bc47826a8"
Nov 01 05:04:26.303499 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:26.303480    5689 scope.go:122] "RemoveContainer" containerID="926cf2e9417e15c25ee90ffb91fa457785273e7901c6f240cbc6763bc47826a8"
Nov 01 05:04:26.303787 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:04:26.303768    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"926cf2e9417e15c25ee90ffb91fa457785273e7901c6f240cbc6763bc47826a8\": container with ID starting with 926cf2e9417e15c25ee90ffb91fa457785273e7901c6f240cbc6763bc47826a8 not found: ID does not exist" containerID="926cf2e9417e15c25ee90ffb91fa457785273e7901c6f240cbc6763bc47826a8"
Nov 01 05:04:26.303833 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:26.303805    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"926cf2e9417e15c25ee90ffb91fa457785273e7901c6f240cbc6763bc47826a8"} err="failed to get container status \"926cf2e9417e15c25ee90ffb91fa457785273e7901c6f240cbc6763bc47826a8\": rpc error: code = NotFound desc = could not find container \"926cf2e9417e15c25ee90ffb91fa457785273e7901c6f240cbc6763bc47826a8\": container with ID starting with 926cf2e9417e15c25ee90ffb91fa457785273e7901c6f240cbc6763bc47826a8 not found: ID does not exist"
Nov 01 05:04:26.456929 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:26.456896    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-t7btw-l6t25"]
Nov 01 05:04:26.506233 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:26.506206    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-t7btw-l6t25"]
Nov 01 05:04:28.065307 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:28.065277    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f4aae57-6429-4f9e-aaa5-c700bc26b4f5" path="/var/lib/kubelet/pods/6f4aae57-6429-4f9e-aaa5-c700bc26b4f5/volumes"
Nov 01 05:04:30.067040 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:04:30.067007    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:04:35.620731 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:35.620694    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:04:35.803568 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:35.803524    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:04:41.523754 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:41.523723    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-677d79bd4b-jt8p8"]
Nov 01 05:04:41.524129 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:41.524001    5689 cpu_manager.go:414] "containerMap: removing container" podUID="6f4aae57-6429-4f9e-aaa5-c700bc26b4f5" containerName="insights-gathering"
Nov 01 05:04:41.524129 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:41.524011    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f4aae57-6429-4f9e-aaa5-c700bc26b4f5" containerName="insights-gathering"
Nov 01 05:04:41.524129 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:41.524025    5689 cpu_manager.go:414] "containerMap: removing container" podUID="1bd1166d-68e9-4035-b920-e42f5bf05d11" containerName="console"
Nov 01 05:04:41.524129 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:41.524030    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bd1166d-68e9-4035-b920-e42f5bf05d11" containerName="console"
Nov 01 05:04:41.524129 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:41.524039    5689 cpu_manager.go:414] "containerMap: removing container" podUID="ca43c15d-cefb-4e78-93aa-a98573da4feb" containerName="console"
Nov 01 05:04:41.524129 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:41.524044    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca43c15d-cefb-4e78-93aa-a98573da4feb" containerName="console"
Nov 01 05:04:41.524129 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:41.524094    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bd1166d-68e9-4035-b920-e42f5bf05d11" containerName="console"
Nov 01 05:04:41.524129 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:41.524102    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca43c15d-cefb-4e78-93aa-a98573da4feb" containerName="console"
Nov 01 05:04:41.524129 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:41.524111    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f4aae57-6429-4f9e-aaa5-c700bc26b4f5" containerName="insights-gathering"
Nov 01 05:04:41.526629 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:41.526611    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-677d79bd4b-jt8p8"
Nov 01 05:04:41.578746 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:41.578719    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Nov 01 05:04:41.578911 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:41.578837    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Nov 01 05:04:41.578911 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:41.578860    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Nov 01 05:04:41.578911 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:41.578880    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Nov 01 05:04:41.628238 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:41.628215    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Nov 01 05:04:41.628847 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:41.628829    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-5dwj4\""
Nov 01 05:04:41.666356 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:41.666321    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c89ad23e-b3d6-417b-8863-1194dc8dc863-secret-telemeter-client\") pod \"telemeter-client-677d79bd4b-jt8p8\" (UID: \"c89ad23e-b3d6-417b-8863-1194dc8dc863\") " pod="openshift-monitoring/telemeter-client-677d79bd4b-jt8p8"
Nov 01 05:04:41.666509 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:41.666365    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c89ad23e-b3d6-417b-8863-1194dc8dc863-federate-client-tls\") pod \"telemeter-client-677d79bd4b-jt8p8\" (UID: \"c89ad23e-b3d6-417b-8863-1194dc8dc863\") " pod="openshift-monitoring/telemeter-client-677d79bd4b-jt8p8"
Nov 01 05:04:41.666509 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:41.666392    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c89ad23e-b3d6-417b-8863-1194dc8dc863-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-677d79bd4b-jt8p8\" (UID: \"c89ad23e-b3d6-417b-8863-1194dc8dc863\") " pod="openshift-monitoring/telemeter-client-677d79bd4b-jt8p8"
Nov 01 05:04:41.666509 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:41.666492    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c89ad23e-b3d6-417b-8863-1194dc8dc863-telemeter-trusted-ca-bundle\") pod \"telemeter-client-677d79bd4b-jt8p8\" (UID: \"c89ad23e-b3d6-417b-8863-1194dc8dc863\") " pod="openshift-monitoring/telemeter-client-677d79bd4b-jt8p8"
Nov 01 05:04:41.666676 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:41.666514    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c89ad23e-b3d6-417b-8863-1194dc8dc863-telemeter-client-tls\") pod \"telemeter-client-677d79bd4b-jt8p8\" (UID: \"c89ad23e-b3d6-417b-8863-1194dc8dc863\") " pod="openshift-monitoring/telemeter-client-677d79bd4b-jt8p8"
Nov 01 05:04:41.666676 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:41.666547    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj6gt\" (UniqueName: \"kubernetes.io/projected/c89ad23e-b3d6-417b-8863-1194dc8dc863-kube-api-access-jj6gt\") pod \"telemeter-client-677d79bd4b-jt8p8\" (UID: \"c89ad23e-b3d6-417b-8863-1194dc8dc863\") " pod="openshift-monitoring/telemeter-client-677d79bd4b-jt8p8"
Nov 01 05:04:41.666676 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:41.666569    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c89ad23e-b3d6-417b-8863-1194dc8dc863-serving-certs-ca-bundle\") pod \"telemeter-client-677d79bd4b-jt8p8\" (UID: \"c89ad23e-b3d6-417b-8863-1194dc8dc863\") " pod="openshift-monitoring/telemeter-client-677d79bd4b-jt8p8"
Nov 01 05:04:41.666676 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:41.666634    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c89ad23e-b3d6-417b-8863-1194dc8dc863-metrics-client-ca\") pod \"telemeter-client-677d79bd4b-jt8p8\" (UID: \"c89ad23e-b3d6-417b-8863-1194dc8dc863\") " pod="openshift-monitoring/telemeter-client-677d79bd4b-jt8p8"
Nov 01 05:04:41.678778 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:41.678760    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Nov 01 05:04:41.679307 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:41.679290    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-677d79bd4b-jt8p8"]
Nov 01 05:04:41.767640 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:41.767612    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c89ad23e-b3d6-417b-8863-1194dc8dc863-secret-telemeter-client\") pod \"telemeter-client-677d79bd4b-jt8p8\" (UID: \"c89ad23e-b3d6-417b-8863-1194dc8dc863\") " pod="openshift-monitoring/telemeter-client-677d79bd4b-jt8p8"
Nov 01 05:04:41.767773 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:41.767650    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c89ad23e-b3d6-417b-8863-1194dc8dc863-federate-client-tls\") pod \"telemeter-client-677d79bd4b-jt8p8\" (UID: \"c89ad23e-b3d6-417b-8863-1194dc8dc863\") " pod="openshift-monitoring/telemeter-client-677d79bd4b-jt8p8"
Nov 01 05:04:41.767773 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:41.767670    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c89ad23e-b3d6-417b-8863-1194dc8dc863-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-677d79bd4b-jt8p8\" (UID: \"c89ad23e-b3d6-417b-8863-1194dc8dc863\") " pod="openshift-monitoring/telemeter-client-677d79bd4b-jt8p8"
Nov 01 05:04:41.767773 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:41.767690    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c89ad23e-b3d6-417b-8863-1194dc8dc863-telemeter-trusted-ca-bundle\") pod \"telemeter-client-677d79bd4b-jt8p8\" (UID: \"c89ad23e-b3d6-417b-8863-1194dc8dc863\") " pod="openshift-monitoring/telemeter-client-677d79bd4b-jt8p8"
Nov 01 05:04:41.767773 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:41.767707    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c89ad23e-b3d6-417b-8863-1194dc8dc863-telemeter-client-tls\") pod \"telemeter-client-677d79bd4b-jt8p8\" (UID: \"c89ad23e-b3d6-417b-8863-1194dc8dc863\") " pod="openshift-monitoring/telemeter-client-677d79bd4b-jt8p8"
Nov 01 05:04:41.767773 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:41.767748    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj6gt\" (UniqueName: \"kubernetes.io/projected/c89ad23e-b3d6-417b-8863-1194dc8dc863-kube-api-access-jj6gt\") pod \"telemeter-client-677d79bd4b-jt8p8\" (UID: \"c89ad23e-b3d6-417b-8863-1194dc8dc863\") " pod="openshift-monitoring/telemeter-client-677d79bd4b-jt8p8"
Nov 01 05:04:41.768084 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:41.767799    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c89ad23e-b3d6-417b-8863-1194dc8dc863-serving-certs-ca-bundle\") pod \"telemeter-client-677d79bd4b-jt8p8\" (UID: \"c89ad23e-b3d6-417b-8863-1194dc8dc863\") " pod="openshift-monitoring/telemeter-client-677d79bd4b-jt8p8"
Nov 01 05:04:41.768084 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:41.767826    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c89ad23e-b3d6-417b-8863-1194dc8dc863-metrics-client-ca\") pod \"telemeter-client-677d79bd4b-jt8p8\" (UID: \"c89ad23e-b3d6-417b-8863-1194dc8dc863\") " pod="openshift-monitoring/telemeter-client-677d79bd4b-jt8p8"
Nov 01 05:04:41.768630 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:41.768530    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c89ad23e-b3d6-417b-8863-1194dc8dc863-metrics-client-ca\") pod \"telemeter-client-677d79bd4b-jt8p8\" (UID: \"c89ad23e-b3d6-417b-8863-1194dc8dc863\") " pod="openshift-monitoring/telemeter-client-677d79bd4b-jt8p8"
Nov 01 05:04:41.768828 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:41.768797    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c89ad23e-b3d6-417b-8863-1194dc8dc863-telemeter-trusted-ca-bundle\") pod \"telemeter-client-677d79bd4b-jt8p8\" (UID: \"c89ad23e-b3d6-417b-8863-1194dc8dc863\") " pod="openshift-monitoring/telemeter-client-677d79bd4b-jt8p8"
Nov 01 05:04:41.768969 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:41.768954    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c89ad23e-b3d6-417b-8863-1194dc8dc863-serving-certs-ca-bundle\") pod \"telemeter-client-677d79bd4b-jt8p8\" (UID: \"c89ad23e-b3d6-417b-8863-1194dc8dc863\") " pod="openshift-monitoring/telemeter-client-677d79bd4b-jt8p8"
Nov 01 05:04:41.770337 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:41.770308    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c89ad23e-b3d6-417b-8863-1194dc8dc863-federate-client-tls\") pod \"telemeter-client-677d79bd4b-jt8p8\" (UID: \"c89ad23e-b3d6-417b-8863-1194dc8dc863\") " pod="openshift-monitoring/telemeter-client-677d79bd4b-jt8p8"
Nov 01 05:04:41.770429 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:41.770398    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c89ad23e-b3d6-417b-8863-1194dc8dc863-secret-telemeter-client\") pod \"telemeter-client-677d79bd4b-jt8p8\" (UID: \"c89ad23e-b3d6-417b-8863-1194dc8dc863\") " pod="openshift-monitoring/telemeter-client-677d79bd4b-jt8p8"
Nov 01 05:04:41.770737 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:41.770720    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c89ad23e-b3d6-417b-8863-1194dc8dc863-telemeter-client-tls\") pod \"telemeter-client-677d79bd4b-jt8p8\" (UID: \"c89ad23e-b3d6-417b-8863-1194dc8dc863\") " pod="openshift-monitoring/telemeter-client-677d79bd4b-jt8p8"
Nov 01 05:04:41.770827 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:41.770806    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c89ad23e-b3d6-417b-8863-1194dc8dc863-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-677d79bd4b-jt8p8\" (UID: \"c89ad23e-b3d6-417b-8863-1194dc8dc863\") " pod="openshift-monitoring/telemeter-client-677d79bd4b-jt8p8"
Nov 01 05:04:41.829008 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:41.828956    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj6gt\" (UniqueName: \"kubernetes.io/projected/c89ad23e-b3d6-417b-8863-1194dc8dc863-kube-api-access-jj6gt\") pod \"telemeter-client-677d79bd4b-jt8p8\" (UID: \"c89ad23e-b3d6-417b-8863-1194dc8dc863\") " pod="openshift-monitoring/telemeter-client-677d79bd4b-jt8p8"
Nov 01 05:04:41.838614 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:41.838583    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-677d79bd4b-jt8p8"
Nov 01 05:04:42.262715 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:42.262684    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-677d79bd4b-jt8p8"]
Nov 01 05:04:42.267009 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:04:42.266980    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc89ad23e_b3d6_417b_8863_1194dc8dc863.slice/crio-027777fea11446d617b42bd5a50f3088a79577aa42d6dd01297a5a884e027e25 WatchSource:0}: Error finding container 027777fea11446d617b42bd5a50f3088a79577aa42d6dd01297a5a884e027e25: Status 404 returned error can't find the container with id 027777fea11446d617b42bd5a50f3088a79577aa42d6dd01297a5a884e027e25
Nov 01 05:04:42.349597 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:42.349566    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-677d79bd4b-jt8p8" event={"ID":"c89ad23e-b3d6-417b-8863-1194dc8dc863","Type":"ContainerStarted","Data":"027777fea11446d617b42bd5a50f3088a79577aa42d6dd01297a5a884e027e25"}
Nov 01 05:04:45.361134 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:45.361103    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-677d79bd4b-jt8p8" event={"ID":"c89ad23e-b3d6-417b-8863-1194dc8dc863","Type":"ContainerStarted","Data":"7f62719bb14a7b49d859db26f643b17fb6f9630391c894d7247b62b2c78fb291"}
Nov 01 05:04:47.369558 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:47.369516    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-677d79bd4b-jt8p8" event={"ID":"c89ad23e-b3d6-417b-8863-1194dc8dc863","Type":"ContainerStarted","Data":"7ff90d658aa5850a2ee5e960cb45eae04fa78c2b978840612cb7e7d3d768e0c5"}
Nov 01 05:04:47.369940 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:47.369589    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-677d79bd4b-jt8p8" event={"ID":"c89ad23e-b3d6-417b-8863-1194dc8dc863","Type":"ContainerStarted","Data":"a871b06ee6d9fb6b0741e5db1e82dc6289ce982fd48712daeb024d5c3efada7e"}
Nov 01 05:04:47.590903 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:47.590856    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-677d79bd4b-jt8p8" podStartSLOduration=2.567947416 podStartE2EDuration="6.590844409s" totalImagesPullingTime="4.022896993s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:04:41 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:04:47.540733823 +0000 UTC m=+65.611530543" watchObservedRunningTime="2026-11-01 05:04:47.590844409 +0000 UTC m=+65.661641130"
Nov 01 05:04:47.976467 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:47.976407    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7cfd88976f-fnm79"]
Nov 01 05:04:47.978604 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:47.978577    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cfd88976f-fnm79"
Nov 01 05:04:48.040797 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:48.040768    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7cfd88976f-fnm79"]
Nov 01 05:04:48.113575 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:48.113530    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a13c36d6-aafb-4e3e-ac88-ef972a4d5617-oauth-serving-cert\") pod \"console-7cfd88976f-fnm79\" (UID: \"a13c36d6-aafb-4e3e-ac88-ef972a4d5617\") " pod="openshift-console/console-7cfd88976f-fnm79"
Nov 01 05:04:48.113575 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:48.113583    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a13c36d6-aafb-4e3e-ac88-ef972a4d5617-service-ca\") pod \"console-7cfd88976f-fnm79\" (UID: \"a13c36d6-aafb-4e3e-ac88-ef972a4d5617\") " pod="openshift-console/console-7cfd88976f-fnm79"
Nov 01 05:04:48.113760 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:48.113680    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a13c36d6-aafb-4e3e-ac88-ef972a4d5617-console-config\") pod \"console-7cfd88976f-fnm79\" (UID: \"a13c36d6-aafb-4e3e-ac88-ef972a4d5617\") " pod="openshift-console/console-7cfd88976f-fnm79"
Nov 01 05:04:48.113760 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:48.113712    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a13c36d6-aafb-4e3e-ac88-ef972a4d5617-tmp\") pod \"console-7cfd88976f-fnm79\" (UID: \"a13c36d6-aafb-4e3e-ac88-ef972a4d5617\") " pod="openshift-console/console-7cfd88976f-fnm79"
Nov 01 05:04:48.113760 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:48.113739    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a13c36d6-aafb-4e3e-ac88-ef972a4d5617-console-oauth-config\") pod \"console-7cfd88976f-fnm79\" (UID: \"a13c36d6-aafb-4e3e-ac88-ef972a4d5617\") " pod="openshift-console/console-7cfd88976f-fnm79"
Nov 01 05:04:48.113911 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:48.113789    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a13c36d6-aafb-4e3e-ac88-ef972a4d5617-trusted-ca-bundle\") pod \"console-7cfd88976f-fnm79\" (UID: \"a13c36d6-aafb-4e3e-ac88-ef972a4d5617\") " pod="openshift-console/console-7cfd88976f-fnm79"
Nov 01 05:04:48.113911 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:48.113857    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a13c36d6-aafb-4e3e-ac88-ef972a4d5617-console-serving-cert\") pod \"console-7cfd88976f-fnm79\" (UID: \"a13c36d6-aafb-4e3e-ac88-ef972a4d5617\") " pod="openshift-console/console-7cfd88976f-fnm79"
Nov 01 05:04:48.114012 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:48.113927    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw54g\" (UniqueName: \"kubernetes.io/projected/a13c36d6-aafb-4e3e-ac88-ef972a4d5617-kube-api-access-fw54g\") pod \"console-7cfd88976f-fnm79\" (UID: \"a13c36d6-aafb-4e3e-ac88-ef972a4d5617\") " pod="openshift-console/console-7cfd88976f-fnm79"
Nov 01 05:04:48.214841 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:48.214807    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a13c36d6-aafb-4e3e-ac88-ef972a4d5617-console-config\") pod \"console-7cfd88976f-fnm79\" (UID: \"a13c36d6-aafb-4e3e-ac88-ef972a4d5617\") " pod="openshift-console/console-7cfd88976f-fnm79"
Nov 01 05:04:48.214841 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:48.214844    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a13c36d6-aafb-4e3e-ac88-ef972a4d5617-tmp\") pod \"console-7cfd88976f-fnm79\" (UID: \"a13c36d6-aafb-4e3e-ac88-ef972a4d5617\") " pod="openshift-console/console-7cfd88976f-fnm79"
Nov 01 05:04:48.215019 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:48.214866    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a13c36d6-aafb-4e3e-ac88-ef972a4d5617-console-oauth-config\") pod \"console-7cfd88976f-fnm79\" (UID: \"a13c36d6-aafb-4e3e-ac88-ef972a4d5617\") " pod="openshift-console/console-7cfd88976f-fnm79"
Nov 01 05:04:48.215019 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:48.214885    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a13c36d6-aafb-4e3e-ac88-ef972a4d5617-trusted-ca-bundle\") pod \"console-7cfd88976f-fnm79\" (UID: \"a13c36d6-aafb-4e3e-ac88-ef972a4d5617\") " pod="openshift-console/console-7cfd88976f-fnm79"
Nov 01 05:04:48.215130 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:48.215015    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a13c36d6-aafb-4e3e-ac88-ef972a4d5617-console-serving-cert\") pod \"console-7cfd88976f-fnm79\" (UID: \"a13c36d6-aafb-4e3e-ac88-ef972a4d5617\") " pod="openshift-console/console-7cfd88976f-fnm79"
Nov 01 05:04:48.215130 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:48.215046    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw54g\" (UniqueName: \"kubernetes.io/projected/a13c36d6-aafb-4e3e-ac88-ef972a4d5617-kube-api-access-fw54g\") pod \"console-7cfd88976f-fnm79\" (UID: \"a13c36d6-aafb-4e3e-ac88-ef972a4d5617\") " pod="openshift-console/console-7cfd88976f-fnm79"
Nov 01 05:04:48.215130 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:48.215082    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a13c36d6-aafb-4e3e-ac88-ef972a4d5617-oauth-serving-cert\") pod \"console-7cfd88976f-fnm79\" (UID: \"a13c36d6-aafb-4e3e-ac88-ef972a4d5617\") " pod="openshift-console/console-7cfd88976f-fnm79"
Nov 01 05:04:48.215130 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:48.215118    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a13c36d6-aafb-4e3e-ac88-ef972a4d5617-service-ca\") pod \"console-7cfd88976f-fnm79\" (UID: \"a13c36d6-aafb-4e3e-ac88-ef972a4d5617\") " pod="openshift-console/console-7cfd88976f-fnm79"
Nov 01 05:04:48.215323 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:48.215233    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a13c36d6-aafb-4e3e-ac88-ef972a4d5617-tmp\") pod \"console-7cfd88976f-fnm79\" (UID: \"a13c36d6-aafb-4e3e-ac88-ef972a4d5617\") " pod="openshift-console/console-7cfd88976f-fnm79"
Nov 01 05:04:48.215813 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:48.215785    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a13c36d6-aafb-4e3e-ac88-ef972a4d5617-trusted-ca-bundle\") pod \"console-7cfd88976f-fnm79\" (UID: \"a13c36d6-aafb-4e3e-ac88-ef972a4d5617\") " pod="openshift-console/console-7cfd88976f-fnm79"
Nov 01 05:04:48.215922 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:48.215874    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a13c36d6-aafb-4e3e-ac88-ef972a4d5617-oauth-serving-cert\") pod \"console-7cfd88976f-fnm79\" (UID: \"a13c36d6-aafb-4e3e-ac88-ef972a4d5617\") " pod="openshift-console/console-7cfd88976f-fnm79"
Nov 01 05:04:48.215922 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:48.215882    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a13c36d6-aafb-4e3e-ac88-ef972a4d5617-service-ca\") pod \"console-7cfd88976f-fnm79\" (UID: \"a13c36d6-aafb-4e3e-ac88-ef972a4d5617\") " pod="openshift-console/console-7cfd88976f-fnm79"
Nov 01 05:04:48.216337 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:48.216295    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a13c36d6-aafb-4e3e-ac88-ef972a4d5617-console-config\") pod \"console-7cfd88976f-fnm79\" (UID: \"a13c36d6-aafb-4e3e-ac88-ef972a4d5617\") " pod="openshift-console/console-7cfd88976f-fnm79"
Nov 01 05:04:48.217532 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:48.217508    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a13c36d6-aafb-4e3e-ac88-ef972a4d5617-console-serving-cert\") pod \"console-7cfd88976f-fnm79\" (UID: \"a13c36d6-aafb-4e3e-ac88-ef972a4d5617\") " pod="openshift-console/console-7cfd88976f-fnm79"
Nov 01 05:04:48.217667 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:48.217647    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a13c36d6-aafb-4e3e-ac88-ef972a4d5617-console-oauth-config\") pod \"console-7cfd88976f-fnm79\" (UID: \"a13c36d6-aafb-4e3e-ac88-ef972a4d5617\") " pod="openshift-console/console-7cfd88976f-fnm79"
Nov 01 05:04:48.275489 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:48.275401    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw54g\" (UniqueName: \"kubernetes.io/projected/a13c36d6-aafb-4e3e-ac88-ef972a4d5617-kube-api-access-fw54g\") pod \"console-7cfd88976f-fnm79\" (UID: \"a13c36d6-aafb-4e3e-ac88-ef972a4d5617\") " pod="openshift-console/console-7cfd88976f-fnm79"
Nov 01 05:04:48.287721 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:48.287699    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:04:48.288633 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:48.288618    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cfd88976f-fnm79"
Nov 01 05:04:48.467613 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:48.467583    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:04:48.658966 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:48.658938    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7cfd88976f-fnm79"]
Nov 01 05:04:48.662637 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:04:48.662609    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda13c36d6_aafb_4e3e_ac88_ef972a4d5617.slice/crio-07d7076dfec98225169d18cc6009b5036215b10dbc5087b0f1466a2fd59b8b26 WatchSource:0}: Error finding container 07d7076dfec98225169d18cc6009b5036215b10dbc5087b0f1466a2fd59b8b26: Status 404 returned error can't find the container with id 07d7076dfec98225169d18cc6009b5036215b10dbc5087b0f1466a2fd59b8b26
Nov 01 05:04:48.683857 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:48.683833    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:04:48.858009 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:48.857973    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:04:49.377664 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:49.377625    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cfd88976f-fnm79" event={"ID":"a13c36d6-aafb-4e3e-ac88-ef972a4d5617","Type":"ContainerStarted","Data":"dddb89600f6b10d83a66dc35770597a0a63f2efa156fca81b91ff366a561d34d"}
Nov 01 05:04:49.377824 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:49.377683    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cfd88976f-fnm79" event={"ID":"a13c36d6-aafb-4e3e-ac88-ef972a4d5617","Type":"ContainerStarted","Data":"07d7076dfec98225169d18cc6009b5036215b10dbc5087b0f1466a2fd59b8b26"}
Nov 01 05:04:49.544186 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:49.544131    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7cfd88976f-fnm79" podStartSLOduration=2.544119157 podStartE2EDuration="2.544119157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:04:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:04:49.494639958 +0000 UTC m=+67.565436677" watchObservedRunningTime="2026-11-01 05:04:49.544119157 +0000 UTC m=+67.614915877"
Nov 01 05:04:53.741938 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:53.741904    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:04:58.289269 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:58.289236    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7cfd88976f-fnm79"
Nov 01 05:04:58.289905 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:58.289292    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7cfd88976f-fnm79"
Nov 01 05:04:58.293989 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:58.293969    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7cfd88976f-fnm79"
Nov 01 05:04:58.410168 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:58.410144    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7cfd88976f-fnm79"
Nov 01 05:04:58.560234 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:04:58.560169    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6877799f6d-xrtpt"]
Nov 01 05:05:00.066591 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:05:00.066558    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:05:23.581250 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:05:23.581148    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-6877799f6d-xrtpt" podUID="e63c480a-630b-4bb6-b4c4-a0f62d781ded" containerName="console" containerID="cri-o://c8f75368b6780c1a35df5bdf008ff4dcb9da2b75a2b6aa1c4e425ea36e0bc7cb" gracePeriod=15
Nov 01 05:05:23.631424 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:05:23.631391    5689 patch_prober.go:29] interesting pod/console-6877799f6d-xrtpt container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.134.0.19:8443/health\": dial tcp 10.134.0.19:8443: connect: connection refused" start-of-body=
Nov 01 05:05:23.631541 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:05:23.631447    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/console-6877799f6d-xrtpt" podUID="e63c480a-630b-4bb6-b4c4-a0f62d781ded" containerName="console" probeResult="failure" output="Get \"https://10.134.0.19:8443/health\": dial tcp 10.134.0.19:8443: connect: connection refused"
Nov 01 05:05:23.827681 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:05:23.827656    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6877799f6d-xrtpt"
Nov 01 05:05:23.861199 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:05:23.861151    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e63c480a-630b-4bb6-b4c4-a0f62d781ded-service-ca\" (UniqueName: \"kubernetes.io/configmap/e63c480a-630b-4bb6-b4c4-a0f62d781ded-service-ca\") pod \"e63c480a-630b-4bb6-b4c4-a0f62d781ded\" (UID: \"e63c480a-630b-4bb6-b4c4-a0f62d781ded\") "
Nov 01 05:05:23.861199 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:05:23.861187    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e63c480a-630b-4bb6-b4c4-a0f62d781ded-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e63c480a-630b-4bb6-b4c4-a0f62d781ded-console-serving-cert\") pod \"e63c480a-630b-4bb6-b4c4-a0f62d781ded\" (UID: \"e63c480a-630b-4bb6-b4c4-a0f62d781ded\") "
Nov 01 05:05:23.861310 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:05:23.861236    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e63c480a-630b-4bb6-b4c4-a0f62d781ded-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e63c480a-630b-4bb6-b4c4-a0f62d781ded-oauth-serving-cert\") pod \"e63c480a-630b-4bb6-b4c4-a0f62d781ded\" (UID: \"e63c480a-630b-4bb6-b4c4-a0f62d781ded\") "
Nov 01 05:05:23.861310 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:05:23.861256    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e63c480a-630b-4bb6-b4c4-a0f62d781ded-tmp\" (UniqueName: \"kubernetes.io/empty-dir/e63c480a-630b-4bb6-b4c4-a0f62d781ded-tmp\") pod \"e63c480a-630b-4bb6-b4c4-a0f62d781ded\" (UID: \"e63c480a-630b-4bb6-b4c4-a0f62d781ded\") "
Nov 01 05:05:23.861310 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:05:23.861283    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e63c480a-630b-4bb6-b4c4-a0f62d781ded-console-config\" (UniqueName: \"kubernetes.io/configmap/e63c480a-630b-4bb6-b4c4-a0f62d781ded-console-config\") pod \"e63c480a-630b-4bb6-b4c4-a0f62d781ded\" (UID: \"e63c480a-630b-4bb6-b4c4-a0f62d781ded\") "
Nov 01 05:05:23.861310 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:05:23.861308    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e63c480a-630b-4bb6-b4c4-a0f62d781ded-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e63c480a-630b-4bb6-b4c4-a0f62d781ded-console-oauth-config\") pod \"e63c480a-630b-4bb6-b4c4-a0f62d781ded\" (UID: \"e63c480a-630b-4bb6-b4c4-a0f62d781ded\") "
Nov 01 05:05:23.861542 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:05:23.861351    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e63c480a-630b-4bb6-b4c4-a0f62d781ded-kube-api-access-nlm6w\" (UniqueName: \"kubernetes.io/projected/e63c480a-630b-4bb6-b4c4-a0f62d781ded-kube-api-access-nlm6w\") pod \"e63c480a-630b-4bb6-b4c4-a0f62d781ded\" (UID: \"e63c480a-630b-4bb6-b4c4-a0f62d781ded\") "
Nov 01 05:05:23.861542 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:05:23.861401    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e63c480a-630b-4bb6-b4c4-a0f62d781ded-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e63c480a-630b-4bb6-b4c4-a0f62d781ded-trusted-ca-bundle\") pod \"e63c480a-630b-4bb6-b4c4-a0f62d781ded\" (UID: \"e63c480a-630b-4bb6-b4c4-a0f62d781ded\") "
Nov 01 05:05:23.861654 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:05:23.861509    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e63c480a-630b-4bb6-b4c4-a0f62d781ded-service-ca" pod "e63c480a-630b-4bb6-b4c4-a0f62d781ded" (UID: "e63c480a-630b-4bb6-b4c4-a0f62d781ded"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:05:23.861654 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:05:23.861644    5689 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e63c480a-630b-4bb6-b4c4-a0f62d781ded-service-ca\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:05:23.861827 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:05:23.861792    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e63c480a-630b-4bb6-b4c4-a0f62d781ded-oauth-serving-cert" pod "e63c480a-630b-4bb6-b4c4-a0f62d781ded" (UID: "e63c480a-630b-4bb6-b4c4-a0f62d781ded"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:05:23.861827 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:05:23.861804    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e63c480a-630b-4bb6-b4c4-a0f62d781ded-tmp" pod "e63c480a-630b-4bb6-b4c4-a0f62d781ded" (UID: "e63c480a-630b-4bb6-b4c4-a0f62d781ded"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:05:23.862026 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:05:23.861945    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e63c480a-630b-4bb6-b4c4-a0f62d781ded-trusted-ca-bundle" pod "e63c480a-630b-4bb6-b4c4-a0f62d781ded" (UID: "e63c480a-630b-4bb6-b4c4-a0f62d781ded"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:05:23.862026 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:05:23.861941    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e63c480a-630b-4bb6-b4c4-a0f62d781ded-console-config" pod "e63c480a-630b-4bb6-b4c4-a0f62d781ded" (UID: "e63c480a-630b-4bb6-b4c4-a0f62d781ded"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:05:23.863570 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:05:23.863547    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e63c480a-630b-4bb6-b4c4-a0f62d781ded-console-oauth-config" pod "e63c480a-630b-4bb6-b4c4-a0f62d781ded" (UID: "e63c480a-630b-4bb6-b4c4-a0f62d781ded"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 05:05:23.863669 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:05:23.863615    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e63c480a-630b-4bb6-b4c4-a0f62d781ded-kube-api-access-nlm6w" pod "e63c480a-630b-4bb6-b4c4-a0f62d781ded" (UID: "e63c480a-630b-4bb6-b4c4-a0f62d781ded"). InnerVolumeSpecName "kube-api-access-nlm6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:05:23.863761 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:05:23.863742    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e63c480a-630b-4bb6-b4c4-a0f62d781ded-console-serving-cert" pod "e63c480a-630b-4bb6-b4c4-a0f62d781ded" (UID: "e63c480a-630b-4bb6-b4c4-a0f62d781ded"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 05:05:23.962593 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:05:23.962564    5689 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e63c480a-630b-4bb6-b4c4-a0f62d781ded-oauth-serving-cert\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:05:23.962593 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:05:23.962587    5689 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e63c480a-630b-4bb6-b4c4-a0f62d781ded-tmp\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:05:23.962593 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:05:23.962597    5689 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e63c480a-630b-4bb6-b4c4-a0f62d781ded-console-config\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:05:23.962789 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:05:23.962605    5689 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e63c480a-630b-4bb6-b4c4-a0f62d781ded-console-oauth-config\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:05:23.962789 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:05:23.962614    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlm6w\" (UniqueName: \"kubernetes.io/projected/e63c480a-630b-4bb6-b4c4-a0f62d781ded-kube-api-access-nlm6w\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:05:23.962789 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:05:23.962624    5689 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e63c480a-630b-4bb6-b4c4-a0f62d781ded-trusted-ca-bundle\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:05:23.962789 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:05:23.962633    5689 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e63c480a-630b-4bb6-b4c4-a0f62d781ded-console-serving-cert\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:05:24.495674 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:05:24.495640    5689 generic.go:356] "Generic (PLEG): container finished" podID="e63c480a-630b-4bb6-b4c4-a0f62d781ded" containerID="c8f75368b6780c1a35df5bdf008ff4dcb9da2b75a2b6aa1c4e425ea36e0bc7cb" exitCode=0
Nov 01 05:05:24.495830 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:05:24.495684    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6877799f6d-xrtpt"
Nov 01 05:05:24.495830 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:05:24.495720    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6877799f6d-xrtpt" event={"ID":"e63c480a-630b-4bb6-b4c4-a0f62d781ded","Type":"ContainerDied","Data":"c8f75368b6780c1a35df5bdf008ff4dcb9da2b75a2b6aa1c4e425ea36e0bc7cb"}
Nov 01 05:05:24.495830 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:05:24.495771    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6877799f6d-xrtpt" event={"ID":"e63c480a-630b-4bb6-b4c4-a0f62d781ded","Type":"ContainerDied","Data":"f82252073384d9ab4935bbd3c186b32e634315b8d1ac5bc19c7ddcf6c21a4adc"}
Nov 01 05:05:24.495830 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:05:24.495791    5689 scope.go:122] "RemoveContainer" containerID="c8f75368b6780c1a35df5bdf008ff4dcb9da2b75a2b6aa1c4e425ea36e0bc7cb"
Nov 01 05:05:24.503585 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:05:24.503570    5689 scope.go:122] "RemoveContainer" containerID="c8f75368b6780c1a35df5bdf008ff4dcb9da2b75a2b6aa1c4e425ea36e0bc7cb"
Nov 01 05:05:24.503852 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:05:24.503834    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8f75368b6780c1a35df5bdf008ff4dcb9da2b75a2b6aa1c4e425ea36e0bc7cb\": container with ID starting with c8f75368b6780c1a35df5bdf008ff4dcb9da2b75a2b6aa1c4e425ea36e0bc7cb not found: ID does not exist" containerID="c8f75368b6780c1a35df5bdf008ff4dcb9da2b75a2b6aa1c4e425ea36e0bc7cb"
Nov 01 05:05:24.503898 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:05:24.503863    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8f75368b6780c1a35df5bdf008ff4dcb9da2b75a2b6aa1c4e425ea36e0bc7cb"} err="failed to get container status \"c8f75368b6780c1a35df5bdf008ff4dcb9da2b75a2b6aa1c4e425ea36e0bc7cb\": rpc error: code = NotFound desc = could not find container \"c8f75368b6780c1a35df5bdf008ff4dcb9da2b75a2b6aa1c4e425ea36e0bc7cb\": container with ID starting with c8f75368b6780c1a35df5bdf008ff4dcb9da2b75a2b6aa1c4e425ea36e0bc7cb not found: ID does not exist"
Nov 01 05:05:24.659053 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:05:24.659020    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6877799f6d-xrtpt"]
Nov 01 05:05:24.708213 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:05:24.708187    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6877799f6d-xrtpt"]
Nov 01 05:05:26.065128 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:05:26.065094    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e63c480a-630b-4bb6-b4c4-a0f62d781ded" path="/var/lib/kubelet/pods/e63c480a-630b-4bb6-b4c4-a0f62d781ded/volumes"
Nov 01 05:05:30.068735 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:05:30.068701    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:05:53.403445 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:05:53.403402    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:05:53.578237 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:05:53.578207    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:05:54.077103 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:05:54.077066    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:05:54.291290 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:05:54.291256    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:05:59.266971 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:05:59.266939    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-796f9c5b75-6c52c"]
Nov 01 05:05:59.267573 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:05:59.267223    5689 cpu_manager.go:414] "containerMap: removing container" podUID="e63c480a-630b-4bb6-b4c4-a0f62d781ded" containerName="console"
Nov 01 05:05:59.267573 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:05:59.267234    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e63c480a-630b-4bb6-b4c4-a0f62d781ded" containerName="console"
Nov 01 05:05:59.267573 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:05:59.267308    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="e63c480a-630b-4bb6-b4c4-a0f62d781ded" containerName="console"
Nov 01 05:05:59.270480 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:05:59.270445    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-796f9c5b75-6c52c"
Nov 01 05:05:59.374884 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:05:59.374854    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-796f9c5b75-6c52c"]
Nov 01 05:05:59.402470 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:05:59.402429    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cfc5db46-4a4b-4287-abec-0ef2e6685fec-console-config\") pod \"console-796f9c5b75-6c52c\" (UID: \"cfc5db46-4a4b-4287-abec-0ef2e6685fec\") " pod="openshift-console/console-796f9c5b75-6c52c"
Nov 01 05:05:59.402594 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:05:59.402475    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfc5db46-4a4b-4287-abec-0ef2e6685fec-trusted-ca-bundle\") pod \"console-796f9c5b75-6c52c\" (UID: \"cfc5db46-4a4b-4287-abec-0ef2e6685fec\") " pod="openshift-console/console-796f9c5b75-6c52c"
Nov 01 05:05:59.402594 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:05:59.402500    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cfc5db46-4a4b-4287-abec-0ef2e6685fec-oauth-serving-cert\") pod \"console-796f9c5b75-6c52c\" (UID: \"cfc5db46-4a4b-4287-abec-0ef2e6685fec\") " pod="openshift-console/console-796f9c5b75-6c52c"
Nov 01 05:05:59.402594 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:05:59.402568    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x85v5\" (UniqueName: \"kubernetes.io/projected/cfc5db46-4a4b-4287-abec-0ef2e6685fec-kube-api-access-x85v5\") pod \"console-796f9c5b75-6c52c\" (UID: \"cfc5db46-4a4b-4287-abec-0ef2e6685fec\") " pod="openshift-console/console-796f9c5b75-6c52c"
Nov 01 05:05:59.402694 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:05:59.402639    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cfc5db46-4a4b-4287-abec-0ef2e6685fec-console-serving-cert\") pod \"console-796f9c5b75-6c52c\" (UID: \"cfc5db46-4a4b-4287-abec-0ef2e6685fec\") " pod="openshift-console/console-796f9c5b75-6c52c"
Nov 01 05:05:59.402694 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:05:59.402661    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cfc5db46-4a4b-4287-abec-0ef2e6685fec-tmp\") pod \"console-796f9c5b75-6c52c\" (UID: \"cfc5db46-4a4b-4287-abec-0ef2e6685fec\") " pod="openshift-console/console-796f9c5b75-6c52c"
Nov 01 05:05:59.402694 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:05:59.402680    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cfc5db46-4a4b-4287-abec-0ef2e6685fec-console-oauth-config\") pod \"console-796f9c5b75-6c52c\" (UID: \"cfc5db46-4a4b-4287-abec-0ef2e6685fec\") " pod="openshift-console/console-796f9c5b75-6c52c"
Nov 01 05:05:59.402784 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:05:59.402757    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cfc5db46-4a4b-4287-abec-0ef2e6685fec-service-ca\") pod \"console-796f9c5b75-6c52c\" (UID: \"cfc5db46-4a4b-4287-abec-0ef2e6685fec\") " pod="openshift-console/console-796f9c5b75-6c52c"
Nov 01 05:05:59.504004 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:05:59.503977    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cfc5db46-4a4b-4287-abec-0ef2e6685fec-service-ca\") pod \"console-796f9c5b75-6c52c\" (UID: \"cfc5db46-4a4b-4287-abec-0ef2e6685fec\") " pod="openshift-console/console-796f9c5b75-6c52c"
Nov 01 05:05:59.504134 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:05:59.504009    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cfc5db46-4a4b-4287-abec-0ef2e6685fec-console-config\") pod \"console-796f9c5b75-6c52c\" (UID: \"cfc5db46-4a4b-4287-abec-0ef2e6685fec\") " pod="openshift-console/console-796f9c5b75-6c52c"
Nov 01 05:05:59.504134 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:05:59.504030    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfc5db46-4a4b-4287-abec-0ef2e6685fec-trusted-ca-bundle\") pod \"console-796f9c5b75-6c52c\" (UID: \"cfc5db46-4a4b-4287-abec-0ef2e6685fec\") " pod="openshift-console/console-796f9c5b75-6c52c"
Nov 01 05:05:59.504134 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:05:59.504053    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cfc5db46-4a4b-4287-abec-0ef2e6685fec-oauth-serving-cert\") pod \"console-796f9c5b75-6c52c\" (UID: \"cfc5db46-4a4b-4287-abec-0ef2e6685fec\") " pod="openshift-console/console-796f9c5b75-6c52c"
Nov 01 05:05:59.504299 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:05:59.504185    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x85v5\" (UniqueName: \"kubernetes.io/projected/cfc5db46-4a4b-4287-abec-0ef2e6685fec-kube-api-access-x85v5\") pod \"console-796f9c5b75-6c52c\" (UID: \"cfc5db46-4a4b-4287-abec-0ef2e6685fec\") " pod="openshift-console/console-796f9c5b75-6c52c"
Nov 01 05:05:59.504299 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:05:59.504243    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cfc5db46-4a4b-4287-abec-0ef2e6685fec-console-serving-cert\") pod \"console-796f9c5b75-6c52c\" (UID: \"cfc5db46-4a4b-4287-abec-0ef2e6685fec\") " pod="openshift-console/console-796f9c5b75-6c52c"
Nov 01 05:05:59.504396 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:05:59.504323    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cfc5db46-4a4b-4287-abec-0ef2e6685fec-tmp\") pod \"console-796f9c5b75-6c52c\" (UID: \"cfc5db46-4a4b-4287-abec-0ef2e6685fec\") " pod="openshift-console/console-796f9c5b75-6c52c"
Nov 01 05:05:59.504396 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:05:59.504371    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cfc5db46-4a4b-4287-abec-0ef2e6685fec-console-oauth-config\") pod \"console-796f9c5b75-6c52c\" (UID: \"cfc5db46-4a4b-4287-abec-0ef2e6685fec\") " pod="openshift-console/console-796f9c5b75-6c52c"
Nov 01 05:05:59.504796 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:05:59.504772    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cfc5db46-4a4b-4287-abec-0ef2e6685fec-tmp\") pod \"console-796f9c5b75-6c52c\" (UID: \"cfc5db46-4a4b-4287-abec-0ef2e6685fec\") " pod="openshift-console/console-796f9c5b75-6c52c"
Nov 01 05:05:59.504893 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:05:59.504798    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cfc5db46-4a4b-4287-abec-0ef2e6685fec-oauth-serving-cert\") pod \"console-796f9c5b75-6c52c\" (UID: \"cfc5db46-4a4b-4287-abec-0ef2e6685fec\") " pod="openshift-console/console-796f9c5b75-6c52c"
Nov 01 05:05:59.504893 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:05:59.504838    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cfc5db46-4a4b-4287-abec-0ef2e6685fec-service-ca\") pod \"console-796f9c5b75-6c52c\" (UID: \"cfc5db46-4a4b-4287-abec-0ef2e6685fec\") " pod="openshift-console/console-796f9c5b75-6c52c"
Nov 01 05:05:59.504893 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:05:59.504857    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cfc5db46-4a4b-4287-abec-0ef2e6685fec-console-config\") pod \"console-796f9c5b75-6c52c\" (UID: \"cfc5db46-4a4b-4287-abec-0ef2e6685fec\") " pod="openshift-console/console-796f9c5b75-6c52c"
Nov 01 05:05:59.505056 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:05:59.505041    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfc5db46-4a4b-4287-abec-0ef2e6685fec-trusted-ca-bundle\") pod \"console-796f9c5b75-6c52c\" (UID: \"cfc5db46-4a4b-4287-abec-0ef2e6685fec\") " pod="openshift-console/console-796f9c5b75-6c52c"
Nov 01 05:05:59.506933 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:05:59.506909    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cfc5db46-4a4b-4287-abec-0ef2e6685fec-console-serving-cert\") pod \"console-796f9c5b75-6c52c\" (UID: \"cfc5db46-4a4b-4287-abec-0ef2e6685fec\") " pod="openshift-console/console-796f9c5b75-6c52c"
Nov 01 05:05:59.507015 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:05:59.507005    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cfc5db46-4a4b-4287-abec-0ef2e6685fec-console-oauth-config\") pod \"console-796f9c5b75-6c52c\" (UID: \"cfc5db46-4a4b-4287-abec-0ef2e6685fec\") " pod="openshift-console/console-796f9c5b75-6c52c"
Nov 01 05:05:59.565118 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:05:59.565067    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x85v5\" (UniqueName: \"kubernetes.io/projected/cfc5db46-4a4b-4287-abec-0ef2e6685fec-kube-api-access-x85v5\") pod \"console-796f9c5b75-6c52c\" (UID: \"cfc5db46-4a4b-4287-abec-0ef2e6685fec\") " pod="openshift-console/console-796f9c5b75-6c52c"
Nov 01 05:05:59.571790 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:05:59.571766    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:05:59.581737 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:05:59.581721    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-796f9c5b75-6c52c"
Nov 01 05:05:59.752329 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:05:59.752296    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:05:59.953150 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:05:59.953117    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-796f9c5b75-6c52c"]
Nov 01 05:05:59.957141 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:05:59.957114    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfc5db46_4a4b_4287_abec_0ef2e6685fec.slice/crio-7a8de0b49e8a86b3d0265c7f28bc84e3ca513bd7fdfd04f15f2ae1d82abcf856 WatchSource:0}: Error finding container 7a8de0b49e8a86b3d0265c7f28bc84e3ca513bd7fdfd04f15f2ae1d82abcf856: Status 404 returned error can't find the container with id 7a8de0b49e8a86b3d0265c7f28bc84e3ca513bd7fdfd04f15f2ae1d82abcf856
Nov 01 05:05:59.966148 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:05:59.966121    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:06:00.070150 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:06:00.070120    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:06:00.145579 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:06:00.145549    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:06:00.619927 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:06:00.619892    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-796f9c5b75-6c52c" event={"ID":"cfc5db46-4a4b-4287-abec-0ef2e6685fec","Type":"ContainerStarted","Data":"416750515e663dd8bfe37ddc9ae26fbc36c599b257bd7623811d6fb71b7da740"}
Nov 01 05:06:00.620295 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:06:00.619969    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-796f9c5b75-6c52c" event={"ID":"cfc5db46-4a4b-4287-abec-0ef2e6685fec","Type":"ContainerStarted","Data":"7a8de0b49e8a86b3d0265c7f28bc84e3ca513bd7fdfd04f15f2ae1d82abcf856"}
Nov 01 05:06:00.737259 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:06:00.737190    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-796f9c5b75-6c52c" podStartSLOduration=1.737177907 podStartE2EDuration="1.737177907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:05:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:06:00.736902462 +0000 UTC m=+138.807699183" watchObservedRunningTime="2026-11-01 05:06:00.737177907 +0000 UTC m=+138.807974638"
Nov 01 05:06:06.165199 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:06:06.165167    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:06:06.342688 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:06:06.342651    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:06:08.731264 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:06:08.731229    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:06:08.916629 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:06:08.916580    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:06:09.582417 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:06:09.582387    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-796f9c5b75-6c52c"
Nov 01 05:06:09.582417 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:06:09.582437    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-796f9c5b75-6c52c"
Nov 01 05:06:09.588188 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:06:09.588167    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-796f9c5b75-6c52c"
Nov 01 05:06:09.653234 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:06:09.653212    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-796f9c5b75-6c52c"
Nov 01 05:06:09.854069 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:06:09.853999    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7cfd88976f-fnm79"]
Nov 01 05:06:15.872286 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:06:15.872244    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:06:30.068122 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:06:30.068088    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:06:34.873380 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:06:34.873323    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7cfd88976f-fnm79" podUID="a13c36d6-aafb-4e3e-ac88-ef972a4d5617" containerName="console" containerID="cri-o://dddb89600f6b10d83a66dc35770597a0a63f2efa156fca81b91ff366a561d34d" gracePeriod=15
Nov 01 05:06:35.120214 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:06:35.120191    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cfd88976f-fnm79"
Nov 01 05:06:35.255926 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:06:35.255878    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a13c36d6-aafb-4e3e-ac88-ef972a4d5617-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a13c36d6-aafb-4e3e-ac88-ef972a4d5617-console-serving-cert\") pod \"a13c36d6-aafb-4e3e-ac88-ef972a4d5617\" (UID: \"a13c36d6-aafb-4e3e-ac88-ef972a4d5617\") "
Nov 01 05:06:35.256119 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:06:35.255987    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a13c36d6-aafb-4e3e-ac88-ef972a4d5617-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a13c36d6-aafb-4e3e-ac88-ef972a4d5617-console-oauth-config\") pod \"a13c36d6-aafb-4e3e-ac88-ef972a4d5617\" (UID: \"a13c36d6-aafb-4e3e-ac88-ef972a4d5617\") "
Nov 01 05:06:35.256119 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:06:35.256021    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a13c36d6-aafb-4e3e-ac88-ef972a4d5617-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a13c36d6-aafb-4e3e-ac88-ef972a4d5617-trusted-ca-bundle\") pod \"a13c36d6-aafb-4e3e-ac88-ef972a4d5617\" (UID: \"a13c36d6-aafb-4e3e-ac88-ef972a4d5617\") "
Nov 01 05:06:35.256119 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:06:35.256056    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a13c36d6-aafb-4e3e-ac88-ef972a4d5617-console-config\" (UniqueName: \"kubernetes.io/configmap/a13c36d6-aafb-4e3e-ac88-ef972a4d5617-console-config\") pod \"a13c36d6-aafb-4e3e-ac88-ef972a4d5617\" (UID: \"a13c36d6-aafb-4e3e-ac88-ef972a4d5617\") "
Nov 01 05:06:35.256119 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:06:35.256099    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a13c36d6-aafb-4e3e-ac88-ef972a4d5617-service-ca\" (UniqueName: \"kubernetes.io/configmap/a13c36d6-aafb-4e3e-ac88-ef972a4d5617-service-ca\") pod \"a13c36d6-aafb-4e3e-ac88-ef972a4d5617\" (UID: \"a13c36d6-aafb-4e3e-ac88-ef972a4d5617\") "
Nov 01 05:06:35.256357 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:06:35.256133    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a13c36d6-aafb-4e3e-ac88-ef972a4d5617-tmp\" (UniqueName: \"kubernetes.io/empty-dir/a13c36d6-aafb-4e3e-ac88-ef972a4d5617-tmp\") pod \"a13c36d6-aafb-4e3e-ac88-ef972a4d5617\" (UID: \"a13c36d6-aafb-4e3e-ac88-ef972a4d5617\") "
Nov 01 05:06:35.256357 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:06:35.256169    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a13c36d6-aafb-4e3e-ac88-ef972a4d5617-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a13c36d6-aafb-4e3e-ac88-ef972a4d5617-oauth-serving-cert\") pod \"a13c36d6-aafb-4e3e-ac88-ef972a4d5617\" (UID: \"a13c36d6-aafb-4e3e-ac88-ef972a4d5617\") "
Nov 01 05:06:35.256357 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:06:35.256199    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a13c36d6-aafb-4e3e-ac88-ef972a4d5617-kube-api-access-fw54g\" (UniqueName: \"kubernetes.io/projected/a13c36d6-aafb-4e3e-ac88-ef972a4d5617-kube-api-access-fw54g\") pod \"a13c36d6-aafb-4e3e-ac88-ef972a4d5617\" (UID: \"a13c36d6-aafb-4e3e-ac88-ef972a4d5617\") "
Nov 01 05:06:35.256637 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:06:35.256599    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a13c36d6-aafb-4e3e-ac88-ef972a4d5617-console-config" pod "a13c36d6-aafb-4e3e-ac88-ef972a4d5617" (UID: "a13c36d6-aafb-4e3e-ac88-ef972a4d5617"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:06:35.256765 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:06:35.256743    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a13c36d6-aafb-4e3e-ac88-ef972a4d5617-tmp" pod "a13c36d6-aafb-4e3e-ac88-ef972a4d5617" (UID: "a13c36d6-aafb-4e3e-ac88-ef972a4d5617"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:06:35.257064 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:06:35.257039    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a13c36d6-aafb-4e3e-ac88-ef972a4d5617-service-ca" pod "a13c36d6-aafb-4e3e-ac88-ef972a4d5617" (UID: "a13c36d6-aafb-4e3e-ac88-ef972a4d5617"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:06:35.257177 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:06:35.257078    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a13c36d6-aafb-4e3e-ac88-ef972a4d5617-oauth-serving-cert" pod "a13c36d6-aafb-4e3e-ac88-ef972a4d5617" (UID: "a13c36d6-aafb-4e3e-ac88-ef972a4d5617"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:06:35.257442 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:06:35.257418    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a13c36d6-aafb-4e3e-ac88-ef972a4d5617-trusted-ca-bundle" pod "a13c36d6-aafb-4e3e-ac88-ef972a4d5617" (UID: "a13c36d6-aafb-4e3e-ac88-ef972a4d5617"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:06:35.258802 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:06:35.258778    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a13c36d6-aafb-4e3e-ac88-ef972a4d5617-console-oauth-config" pod "a13c36d6-aafb-4e3e-ac88-ef972a4d5617" (UID: "a13c36d6-aafb-4e3e-ac88-ef972a4d5617"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 05:06:35.258910 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:06:35.258837    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a13c36d6-aafb-4e3e-ac88-ef972a4d5617-console-serving-cert" pod "a13c36d6-aafb-4e3e-ac88-ef972a4d5617" (UID: "a13c36d6-aafb-4e3e-ac88-ef972a4d5617"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 05:06:35.259241 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:06:35.259218    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a13c36d6-aafb-4e3e-ac88-ef972a4d5617-kube-api-access-fw54g" pod "a13c36d6-aafb-4e3e-ac88-ef972a4d5617" (UID: "a13c36d6-aafb-4e3e-ac88-ef972a4d5617"). InnerVolumeSpecName "kube-api-access-fw54g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:06:35.357690 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:06:35.357662    5689 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a13c36d6-aafb-4e3e-ac88-ef972a4d5617-service-ca\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:06:35.357690 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:06:35.357687    5689 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a13c36d6-aafb-4e3e-ac88-ef972a4d5617-tmp\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:06:35.357843 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:06:35.357699    5689 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a13c36d6-aafb-4e3e-ac88-ef972a4d5617-oauth-serving-cert\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:06:35.357843 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:06:35.357708    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fw54g\" (UniqueName: \"kubernetes.io/projected/a13c36d6-aafb-4e3e-ac88-ef972a4d5617-kube-api-access-fw54g\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:06:35.357843 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:06:35.357718    5689 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a13c36d6-aafb-4e3e-ac88-ef972a4d5617-console-serving-cert\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:06:35.357843 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:06:35.357726    5689 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a13c36d6-aafb-4e3e-ac88-ef972a4d5617-console-oauth-config\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:06:35.357843 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:06:35.357734    5689 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a13c36d6-aafb-4e3e-ac88-ef972a4d5617-trusted-ca-bundle\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:06:35.357843 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:06:35.357741    5689 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a13c36d6-aafb-4e3e-ac88-ef972a4d5617-console-config\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:06:35.743610 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:06:35.743576    5689 generic.go:356] "Generic (PLEG): container finished" podID="a13c36d6-aafb-4e3e-ac88-ef972a4d5617" containerID="dddb89600f6b10d83a66dc35770597a0a63f2efa156fca81b91ff366a561d34d" exitCode=0
Nov 01 05:06:35.743763 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:06:35.743621    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cfd88976f-fnm79" event={"ID":"a13c36d6-aafb-4e3e-ac88-ef972a4d5617","Type":"ContainerDied","Data":"dddb89600f6b10d83a66dc35770597a0a63f2efa156fca81b91ff366a561d34d"}
Nov 01 05:06:35.743763 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:06:35.743631    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cfd88976f-fnm79"
Nov 01 05:06:35.743763 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:06:35.743655    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cfd88976f-fnm79" event={"ID":"a13c36d6-aafb-4e3e-ac88-ef972a4d5617","Type":"ContainerDied","Data":"07d7076dfec98225169d18cc6009b5036215b10dbc5087b0f1466a2fd59b8b26"}
Nov 01 05:06:35.743763 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:06:35.743674    5689 scope.go:122] "RemoveContainer" containerID="dddb89600f6b10d83a66dc35770597a0a63f2efa156fca81b91ff366a561d34d"
Nov 01 05:06:35.751832 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:06:35.751808    5689 scope.go:122] "RemoveContainer" containerID="dddb89600f6b10d83a66dc35770597a0a63f2efa156fca81b91ff366a561d34d"
Nov 01 05:06:35.752116 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:06:35.752098    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dddb89600f6b10d83a66dc35770597a0a63f2efa156fca81b91ff366a561d34d\": container with ID starting with dddb89600f6b10d83a66dc35770597a0a63f2efa156fca81b91ff366a561d34d not found: ID does not exist" containerID="dddb89600f6b10d83a66dc35770597a0a63f2efa156fca81b91ff366a561d34d"
Nov 01 05:06:35.752161 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:06:35.752129    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dddb89600f6b10d83a66dc35770597a0a63f2efa156fca81b91ff366a561d34d"} err="failed to get container status \"dddb89600f6b10d83a66dc35770597a0a63f2efa156fca81b91ff366a561d34d\": rpc error: code = NotFound desc = could not find container \"dddb89600f6b10d83a66dc35770597a0a63f2efa156fca81b91ff366a561d34d\": container with ID starting with dddb89600f6b10d83a66dc35770597a0a63f2efa156fca81b91ff366a561d34d not found: ID does not exist"
Nov 01 05:06:35.906069 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:06:35.906039    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7cfd88976f-fnm79"]
Nov 01 05:06:35.956161 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:06:35.956134    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7cfd88976f-fnm79"]
Nov 01 05:06:36.064852 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:06:36.064791    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a13c36d6-aafb-4e3e-ac88-ef972a4d5617" path="/var/lib/kubelet/pods/a13c36d6-aafb-4e3e-ac88-ef972a4d5617/volumes"
Nov 01 05:06:54.445260 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:06:54.445190    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:07:00.066560 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:07:00.066527    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:07:30.069691 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:07:30.069656    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:07:43.710859 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:07:43.710819    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:07:45.349663 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:07:45.349636    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-8f57g"]
Nov 01 05:07:45.350229 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:07:45.350052    5689 cpu_manager.go:414] "containerMap: removing container" podUID="a13c36d6-aafb-4e3e-ac88-ef972a4d5617" containerName="console"
Nov 01 05:07:45.350229 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:07:45.350070    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a13c36d6-aafb-4e3e-ac88-ef972a4d5617" containerName="console"
Nov 01 05:07:45.350229 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:07:45.350151    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="a13c36d6-aafb-4e3e-ac88-ef972a4d5617" containerName="console"
Nov 01 05:07:45.353064 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:07:45.353045    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8f57g"
Nov 01 05:07:45.405637 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:07:45.405608    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 01 05:07:45.405637 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:07:45.405632    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 01 05:07:45.405793 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:07:45.405658    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 01 05:07:45.405793 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:07:45.405681    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 01 05:07:45.423023 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:07:45.422997    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2a0b9b72-9a54-4977-87eb-7a341c1156ea-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-8f57g\" (UID: \"2a0b9b72-9a54-4977-87eb-7a341c1156ea\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8f57g"
Nov 01 05:07:45.423149 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:07:45.423033    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cq6f\" (UniqueName: \"kubernetes.io/projected/2a0b9b72-9a54-4977-87eb-7a341c1156ea-kube-api-access-2cq6f\") pod \"istiod-openshift-gateway-7876bfcb74-8f57g\" (UID: \"2a0b9b72-9a54-4977-87eb-7a341c1156ea\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8f57g"
Nov 01 05:07:45.423149 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:07:45.423061    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2a0b9b72-9a54-4977-87eb-7a341c1156ea-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-8f57g\" (UID: \"2a0b9b72-9a54-4977-87eb-7a341c1156ea\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8f57g"
Nov 01 05:07:45.423266 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:07:45.423222    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a0b9b72-9a54-4977-87eb-7a341c1156ea-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-8f57g\" (UID: \"2a0b9b72-9a54-4977-87eb-7a341c1156ea\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8f57g"
Nov 01 05:07:45.423328 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:07:45.423281    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2a0b9b72-9a54-4977-87eb-7a341c1156ea-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-8f57g\" (UID: \"2a0b9b72-9a54-4977-87eb-7a341c1156ea\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8f57g"
Nov 01 05:07:45.423328 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:07:45.423315    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a0b9b72-9a54-4977-87eb-7a341c1156ea-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-8f57g\" (UID: \"2a0b9b72-9a54-4977-87eb-7a341c1156ea\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8f57g"
Nov 01 05:07:45.423424 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:07:45.423342    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2a0b9b72-9a54-4977-87eb-7a341c1156ea-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-8f57g\" (UID: \"2a0b9b72-9a54-4977-87eb-7a341c1156ea\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8f57g"
Nov 01 05:07:45.454782 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:07:45.454760    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 01 05:07:45.454782 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:07:45.454767    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 01 05:07:45.454941 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:07:45.454868    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-44fmh\""
Nov 01 05:07:45.455795 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:07:45.455778    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-8f57g"]
Nov 01 05:07:45.524135 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:07:45.524111    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2a0b9b72-9a54-4977-87eb-7a341c1156ea-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-8f57g\" (UID: \"2a0b9b72-9a54-4977-87eb-7a341c1156ea\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8f57g"
Nov 01 05:07:45.524241 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:07:45.524142    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a0b9b72-9a54-4977-87eb-7a341c1156ea-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-8f57g\" (UID: \"2a0b9b72-9a54-4977-87eb-7a341c1156ea\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8f57g"
Nov 01 05:07:45.524241 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:07:45.524175    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2a0b9b72-9a54-4977-87eb-7a341c1156ea-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-8f57g\" (UID: \"2a0b9b72-9a54-4977-87eb-7a341c1156ea\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8f57g"
Nov 01 05:07:45.524241 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:07:45.524200    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a0b9b72-9a54-4977-87eb-7a341c1156ea-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-8f57g\" (UID: \"2a0b9b72-9a54-4977-87eb-7a341c1156ea\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8f57g"
Nov 01 05:07:45.524241 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:07:45.524227    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2a0b9b72-9a54-4977-87eb-7a341c1156ea-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-8f57g\" (UID: \"2a0b9b72-9a54-4977-87eb-7a341c1156ea\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8f57g"
Nov 01 05:07:45.524435 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:07:45.524276    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2a0b9b72-9a54-4977-87eb-7a341c1156ea-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-8f57g\" (UID: \"2a0b9b72-9a54-4977-87eb-7a341c1156ea\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8f57g"
Nov 01 05:07:45.524435 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:07:45.524387    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cq6f\" (UniqueName: \"kubernetes.io/projected/2a0b9b72-9a54-4977-87eb-7a341c1156ea-kube-api-access-2cq6f\") pod \"istiod-openshift-gateway-7876bfcb74-8f57g\" (UID: \"2a0b9b72-9a54-4977-87eb-7a341c1156ea\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8f57g"
Nov 01 05:07:45.524884 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:07:45.524855    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2a0b9b72-9a54-4977-87eb-7a341c1156ea-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-8f57g\" (UID: \"2a0b9b72-9a54-4977-87eb-7a341c1156ea\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8f57g"
Nov 01 05:07:45.526627 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:07:45.526603    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2a0b9b72-9a54-4977-87eb-7a341c1156ea-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-8f57g\" (UID: \"2a0b9b72-9a54-4977-87eb-7a341c1156ea\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8f57g"
Nov 01 05:07:45.526719 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:07:45.526645    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2a0b9b72-9a54-4977-87eb-7a341c1156ea-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-8f57g\" (UID: \"2a0b9b72-9a54-4977-87eb-7a341c1156ea\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8f57g"
Nov 01 05:07:45.526719 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:07:45.526665    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a0b9b72-9a54-4977-87eb-7a341c1156ea-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-8f57g\" (UID: \"2a0b9b72-9a54-4977-87eb-7a341c1156ea\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8f57g"
Nov 01 05:07:45.526832 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:07:45.526733    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2a0b9b72-9a54-4977-87eb-7a341c1156ea-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-8f57g\" (UID: \"2a0b9b72-9a54-4977-87eb-7a341c1156ea\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8f57g"
Nov 01 05:07:45.585122 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:07:45.585087    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a0b9b72-9a54-4977-87eb-7a341c1156ea-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-8f57g\" (UID: \"2a0b9b72-9a54-4977-87eb-7a341c1156ea\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8f57g"
Nov 01 05:07:45.585236 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:07:45.585172    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cq6f\" (UniqueName: \"kubernetes.io/projected/2a0b9b72-9a54-4977-87eb-7a341c1156ea-kube-api-access-2cq6f\") pod \"istiod-openshift-gateway-7876bfcb74-8f57g\" (UID: \"2a0b9b72-9a54-4977-87eb-7a341c1156ea\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8f57g"
Nov 01 05:07:45.664236 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:07:45.664211    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8f57g"
Nov 01 05:07:45.849662 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:07:45.849628    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:07:46.088174 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:07:46.088096    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-8f57g"]
Nov 01 05:07:46.091892 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:07:46.091865    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a0b9b72_9a54_4977_87eb_7a341c1156ea.slice/crio-c7efe99a65ea7e18a2af48691ea94ca0242a88b53a344f5481d558032f696979 WatchSource:0}: Error finding container c7efe99a65ea7e18a2af48691ea94ca0242a88b53a344f5481d558032f696979: Status 404 returned error can't find the container with id c7efe99a65ea7e18a2af48691ea94ca0242a88b53a344f5481d558032f696979
Nov 01 05:07:46.768698 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:07:46.768670    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:07:46.979916 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:07:46.979886    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8f57g" event={"ID":"2a0b9b72-9a54-4977-87eb-7a341c1156ea","Type":"ContainerStarted","Data":"c7efe99a65ea7e18a2af48691ea94ca0242a88b53a344f5481d558032f696979"}
Nov 01 05:07:50.074338 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:07:50.074309    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:07:50.074604 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:07:50.074371    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:07:50.995525 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:07:50.995486    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8f57g" event={"ID":"2a0b9b72-9a54-4977-87eb-7a341c1156ea","Type":"ContainerStarted","Data":"44ccba72ba73cad0027a6892995c415b8a1db65b0144ffb482014f1d5061349a"}
Nov 01 05:07:50.995685 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:07:50.995613    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8f57g"
Nov 01 05:07:51.213925 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:07:51.213870    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8f57g" podStartSLOduration=2.233922978 podStartE2EDuration="6.213857841s" totalImagesPullingTime="3.979934863s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:07:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:07:51.16421906 +0000 UTC m=+249.235015796" watchObservedRunningTime="2026-11-01 05:07:51.213857841 +0000 UTC m=+249.284654561"
Nov 01 05:07:51.562954 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:07:51.562917    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:07:51.740822 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:07:51.740779    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:07:52.001111 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:07:52.001081    5689 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-8f57g container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 05:07:52.001274 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:07:52.001131    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8f57g" podUID="2a0b9b72-9a54-4977-87eb-7a341c1156ea" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 05:07:52.983696 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:07:52.983668    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:07:53.268413 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:07:53.268374    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:07:53.444187 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:07:53.444151    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:07:54.782951 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:07:54.782918    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:07:54.999948 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:07:54.999922    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8f57g"
Nov 01 05:07:57.090649 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:07:57.090611    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:08:00.067724 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:08:00.067689    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:08:13.537829 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:08:13.537793    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:08:13.707042 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:08:13.707010    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:08:30.068076 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:08:30.067999    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:08:33.920980 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:08:33.920941    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:08:34.115426 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:08:34.115375    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:08:34.548296 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:08:34.548253    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:08:34.849688 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:08:34.849611    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:08:35.045032 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:08:35.044987    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:08:35.082309 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:08:35.082286    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-85d75859fc-xrxx5"]
Nov 01 05:08:35.086114 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:08:35.086096    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85d75859fc-xrxx5"
Nov 01 05:08:35.186748 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:08:35.186716    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25z48\" (UniqueName: \"kubernetes.io/projected/6ae5462d-1b8c-4328-93f2-22f2a2b1ce21-kube-api-access-25z48\") pod \"opendatahub-operator-controller-manager-85d75859fc-xrxx5\" (UID: \"6ae5462d-1b8c-4328-93f2-22f2a2b1ce21\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85d75859fc-xrxx5"
Nov 01 05:08:35.186901 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:08:35.186759    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6ae5462d-1b8c-4328-93f2-22f2a2b1ce21-manifests\") pod \"opendatahub-operator-controller-manager-85d75859fc-xrxx5\" (UID: \"6ae5462d-1b8c-4328-93f2-22f2a2b1ce21\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85d75859fc-xrxx5"
Nov 01 05:08:35.186901 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:08:35.186841    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6ae5462d-1b8c-4328-93f2-22f2a2b1ce21-tmp\") pod \"opendatahub-operator-controller-manager-85d75859fc-xrxx5\" (UID: \"6ae5462d-1b8c-4328-93f2-22f2a2b1ce21\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85d75859fc-xrxx5"
Nov 01 05:08:35.187011 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:08:35.186908    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6ae5462d-1b8c-4328-93f2-22f2a2b1ce21-apiservice-cert\") pod \"opendatahub-operator-controller-manager-85d75859fc-xrxx5\" (UID: \"6ae5462d-1b8c-4328-93f2-22f2a2b1ce21\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85d75859fc-xrxx5"
Nov 01 05:08:35.187011 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:08:35.186937    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6ae5462d-1b8c-4328-93f2-22f2a2b1ce21-metrics-cert\") pod \"opendatahub-operator-controller-manager-85d75859fc-xrxx5\" (UID: \"6ae5462d-1b8c-4328-93f2-22f2a2b1ce21\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85d75859fc-xrxx5"
Nov 01 05:08:35.187085 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:08:35.187064    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6ae5462d-1b8c-4328-93f2-22f2a2b1ce21-webhook-cert\") pod \"opendatahub-operator-controller-manager-85d75859fc-xrxx5\" (UID: \"6ae5462d-1b8c-4328-93f2-22f2a2b1ce21\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85d75859fc-xrxx5"
Nov 01 05:08:35.211827 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:08:35.211806    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 01 05:08:35.211910 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:08:35.211837    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 01 05:08:35.211978 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:08:35.211962    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 01 05:08:35.212050 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:08:35.212032    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-75h7v\""
Nov 01 05:08:35.212107 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:08:35.212035    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 01 05:08:35.212107 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:08:35.212068    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 01 05:08:35.227227 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:08:35.227200    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:08:35.287600 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:08:35.287577    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6ae5462d-1b8c-4328-93f2-22f2a2b1ce21-webhook-cert\") pod \"opendatahub-operator-controller-manager-85d75859fc-xrxx5\" (UID: \"6ae5462d-1b8c-4328-93f2-22f2a2b1ce21\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85d75859fc-xrxx5"
Nov 01 05:08:35.287736 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:08:35.287617    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25z48\" (UniqueName: \"kubernetes.io/projected/6ae5462d-1b8c-4328-93f2-22f2a2b1ce21-kube-api-access-25z48\") pod \"opendatahub-operator-controller-manager-85d75859fc-xrxx5\" (UID: \"6ae5462d-1b8c-4328-93f2-22f2a2b1ce21\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85d75859fc-xrxx5"
Nov 01 05:08:35.287736 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:08:35.287649    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6ae5462d-1b8c-4328-93f2-22f2a2b1ce21-manifests\") pod \"opendatahub-operator-controller-manager-85d75859fc-xrxx5\" (UID: \"6ae5462d-1b8c-4328-93f2-22f2a2b1ce21\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85d75859fc-xrxx5"
Nov 01 05:08:35.287736 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:08:35.287680    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6ae5462d-1b8c-4328-93f2-22f2a2b1ce21-tmp\") pod \"opendatahub-operator-controller-manager-85d75859fc-xrxx5\" (UID: \"6ae5462d-1b8c-4328-93f2-22f2a2b1ce21\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85d75859fc-xrxx5"
Nov 01 05:08:35.287736 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:08:35.287709    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6ae5462d-1b8c-4328-93f2-22f2a2b1ce21-apiservice-cert\") pod \"opendatahub-operator-controller-manager-85d75859fc-xrxx5\" (UID: \"6ae5462d-1b8c-4328-93f2-22f2a2b1ce21\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85d75859fc-xrxx5"
Nov 01 05:08:35.287736 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:08:35.287733    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6ae5462d-1b8c-4328-93f2-22f2a2b1ce21-metrics-cert\") pod \"opendatahub-operator-controller-manager-85d75859fc-xrxx5\" (UID: \"6ae5462d-1b8c-4328-93f2-22f2a2b1ce21\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85d75859fc-xrxx5"
Nov 01 05:08:35.288109 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:08:35.288087    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6ae5462d-1b8c-4328-93f2-22f2a2b1ce21-manifests\") pod \"opendatahub-operator-controller-manager-85d75859fc-xrxx5\" (UID: \"6ae5462d-1b8c-4328-93f2-22f2a2b1ce21\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85d75859fc-xrxx5"
Nov 01 05:08:35.290131 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:08:35.290101    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6ae5462d-1b8c-4328-93f2-22f2a2b1ce21-tmp\") pod \"opendatahub-operator-controller-manager-85d75859fc-xrxx5\" (UID: \"6ae5462d-1b8c-4328-93f2-22f2a2b1ce21\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85d75859fc-xrxx5"
Nov 01 05:08:35.290131 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:08:35.290115    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6ae5462d-1b8c-4328-93f2-22f2a2b1ce21-webhook-cert\") pod \"opendatahub-operator-controller-manager-85d75859fc-xrxx5\" (UID: \"6ae5462d-1b8c-4328-93f2-22f2a2b1ce21\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85d75859fc-xrxx5"
Nov 01 05:08:35.290308 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:08:35.290153    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6ae5462d-1b8c-4328-93f2-22f2a2b1ce21-metrics-cert\") pod \"opendatahub-operator-controller-manager-85d75859fc-xrxx5\" (UID: \"6ae5462d-1b8c-4328-93f2-22f2a2b1ce21\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85d75859fc-xrxx5"
Nov 01 05:08:35.290308 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:08:35.290248    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6ae5462d-1b8c-4328-93f2-22f2a2b1ce21-apiservice-cert\") pod \"opendatahub-operator-controller-manager-85d75859fc-xrxx5\" (UID: \"6ae5462d-1b8c-4328-93f2-22f2a2b1ce21\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85d75859fc-xrxx5"
Nov 01 05:08:35.335494 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:08:35.335406    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-85d75859fc-xrxx5"]
Nov 01 05:08:35.348758 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:08:35.348734    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25z48\" (UniqueName: \"kubernetes.io/projected/6ae5462d-1b8c-4328-93f2-22f2a2b1ce21-kube-api-access-25z48\") pod \"opendatahub-operator-controller-manager-85d75859fc-xrxx5\" (UID: \"6ae5462d-1b8c-4328-93f2-22f2a2b1ce21\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85d75859fc-xrxx5"
Nov 01 05:08:35.398482 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:08:35.398438    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85d75859fc-xrxx5"
Nov 01 05:08:35.455063 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:08:35.454889    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:08:35.898344 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:08:35.898315    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-85d75859fc-xrxx5"]
Nov 01 05:08:35.901875 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:08:35.901846    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ae5462d_1b8c_4328_93f2_22f2a2b1ce21.slice/crio-3e25426562b1b2301b61c50c38271caf2a67df9c237886518032fe52ced58470 WatchSource:0}: Error finding container 3e25426562b1b2301b61c50c38271caf2a67df9c237886518032fe52ced58470: Status 404 returned error can't find the container with id 3e25426562b1b2301b61c50c38271caf2a67df9c237886518032fe52ced58470
Nov 01 05:08:36.152054 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:08:36.151980    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85d75859fc-xrxx5" event={"ID":"6ae5462d-1b8c-4328-93f2-22f2a2b1ce21","Type":"ContainerStarted","Data":"3e25426562b1b2301b61c50c38271caf2a67df9c237886518032fe52ced58470"}
Nov 01 05:08:36.949364 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:08:36.949329    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:08:37.192385 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:08:37.192324    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:08:38.320421 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:08:38.320362    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:08:38.506212 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:08:38.506181    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:08:40.175474 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:08:40.175423    5689 generic.go:356] "Generic (PLEG): container finished" podID="6ae5462d-1b8c-4328-93f2-22f2a2b1ce21" containerID="b836dba860abd2c81f4d409cb2b167ea5f580b304f0b1ab455e2d2c4fa67532c" exitCode=0
Nov 01 05:08:40.175939 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:08:40.175488    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85d75859fc-xrxx5" event={"ID":"6ae5462d-1b8c-4328-93f2-22f2a2b1ce21","Type":"ContainerDied","Data":"b836dba860abd2c81f4d409cb2b167ea5f580b304f0b1ab455e2d2c4fa67532c"}
Nov 01 05:08:41.181037 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:08:41.181007    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85d75859fc-xrxx5" event={"ID":"6ae5462d-1b8c-4328-93f2-22f2a2b1ce21","Type":"ContainerStarted","Data":"8cbe8dd23debc1a2ea8b339e99a3fff39c8d3c830de500e6975e755b36e91976"}
Nov 01 05:08:41.181417 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:08:41.181120    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85d75859fc-xrxx5"
Nov 01 05:08:41.418929 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:08:41.418868    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85d75859fc-xrxx5" podStartSLOduration=2.833493357 podStartE2EDuration="7.418852666s" totalImagesPullingTime="4.17803249s" totalInitContainerRuntime="407.326819ms" isStatefulPod=false podCreationTimestamp="2026-11-01 05:08:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:08:41.369295558 +0000 UTC m=+299.440092301" watchObservedRunningTime="2026-11-01 05:08:41.418852666 +0000 UTC m=+299.489649386"
Nov 01 05:08:43.166199 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:08:43.166164    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:08:43.342798 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:08:43.342762    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:08:44.309444 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:08:44.309409    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:08:47.115826 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:08:47.115794    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:08:48.316317 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:08:48.316280    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:08:51.185631 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:08:51.185596    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:08:52.120984 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:08:52.120943    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:08:52.186072 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:08:52.186049    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85d75859fc-xrxx5"
Nov 01 05:08:52.300278 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:08:52.300248    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:08:53.349960 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:08:53.349927    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:08:53.910873 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:08:53.910837    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:08:54.491187 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:08:54.491148    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:08:54.728130 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:08:54.728095    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:08:55.163860 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:08:55.163821    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:08:55.346240 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:08:55.346212    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:08:56.834829 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:08:56.834784    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:08:57.314554 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:08:57.314514    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:08:58.760482 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:08:58.760428    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:09:00.068402 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:09:00.068368    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:09:00.904963 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:00.904924    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:09:01.089409 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:01.089378    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:09:01.278051 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:01.277964    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:09:02.973998 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:02.973962    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:09:03.218921 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:03.218886    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:09:03.418926 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:03.418898    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:09:04.417408 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:04.417378    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:09:04.629051 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:04.629017    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:09:05.231422 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:05.231382    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:09:05.673196 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:05.673168    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:09:05.855611 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:05.855580    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:09:06.291772 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:06.291743    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:09:08.000337 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:08.000304    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:09:08.006665 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:08.006640    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5f68b7bd84-5rtgt"]
Nov 01 05:09:08.010085 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:08.010063    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5f68b7bd84-5rtgt"
Nov 01 05:09:08.114014 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:08.113987    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Nov 01 05:09:08.114014 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:08.114011    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Nov 01 05:09:08.114194 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:08.114087    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-ql6nz\""
Nov 01 05:09:08.117931 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:08.117912    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5f68b7bd84-5rtgt"]
Nov 01 05:09:08.145193 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:08.145168    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/93b41328-171b-43ba-9a18-370d66e0cd5d-tmp\") pod \"kube-auth-proxy-5f68b7bd84-5rtgt\" (UID: \"93b41328-171b-43ba-9a18-370d66e0cd5d\") " pod="openshift-ingress/kube-auth-proxy-5f68b7bd84-5rtgt"
Nov 01 05:09:08.145311 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:08.145195    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93b41328-171b-43ba-9a18-370d66e0cd5d-tls-certs\") pod \"kube-auth-proxy-5f68b7bd84-5rtgt\" (UID: \"93b41328-171b-43ba-9a18-370d66e0cd5d\") " pod="openshift-ingress/kube-auth-proxy-5f68b7bd84-5rtgt"
Nov 01 05:09:08.145311 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:08.145222    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjwzk\" (UniqueName: \"kubernetes.io/projected/93b41328-171b-43ba-9a18-370d66e0cd5d-kube-api-access-fjwzk\") pod \"kube-auth-proxy-5f68b7bd84-5rtgt\" (UID: \"93b41328-171b-43ba-9a18-370d66e0cd5d\") " pod="openshift-ingress/kube-auth-proxy-5f68b7bd84-5rtgt"
Nov 01 05:09:08.180059 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:08.180033    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:09:08.245687 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:08.245658    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/93b41328-171b-43ba-9a18-370d66e0cd5d-tmp\") pod \"kube-auth-proxy-5f68b7bd84-5rtgt\" (UID: \"93b41328-171b-43ba-9a18-370d66e0cd5d\") " pod="openshift-ingress/kube-auth-proxy-5f68b7bd84-5rtgt"
Nov 01 05:09:08.245687 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:08.245689    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93b41328-171b-43ba-9a18-370d66e0cd5d-tls-certs\") pod \"kube-auth-proxy-5f68b7bd84-5rtgt\" (UID: \"93b41328-171b-43ba-9a18-370d66e0cd5d\") " pod="openshift-ingress/kube-auth-proxy-5f68b7bd84-5rtgt"
Nov 01 05:09:08.245881 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:08.245713    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjwzk\" (UniqueName: \"kubernetes.io/projected/93b41328-171b-43ba-9a18-370d66e0cd5d-kube-api-access-fjwzk\") pod \"kube-auth-proxy-5f68b7bd84-5rtgt\" (UID: \"93b41328-171b-43ba-9a18-370d66e0cd5d\") " pod="openshift-ingress/kube-auth-proxy-5f68b7bd84-5rtgt"
Nov 01 05:09:08.247954 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:08.247925    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/93b41328-171b-43ba-9a18-370d66e0cd5d-tmp\") pod \"kube-auth-proxy-5f68b7bd84-5rtgt\" (UID: \"93b41328-171b-43ba-9a18-370d66e0cd5d\") " pod="openshift-ingress/kube-auth-proxy-5f68b7bd84-5rtgt"
Nov 01 05:09:08.248104 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:08.248088    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93b41328-171b-43ba-9a18-370d66e0cd5d-tls-certs\") pod \"kube-auth-proxy-5f68b7bd84-5rtgt\" (UID: \"93b41328-171b-43ba-9a18-370d66e0cd5d\") " pod="openshift-ingress/kube-auth-proxy-5f68b7bd84-5rtgt"
Nov 01 05:09:08.306372 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:08.306310    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjwzk\" (UniqueName: \"kubernetes.io/projected/93b41328-171b-43ba-9a18-370d66e0cd5d-kube-api-access-fjwzk\") pod \"kube-auth-proxy-5f68b7bd84-5rtgt\" (UID: \"93b41328-171b-43ba-9a18-370d66e0cd5d\") " pod="openshift-ingress/kube-auth-proxy-5f68b7bd84-5rtgt"
Nov 01 05:09:08.322383 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:08.322346    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5f68b7bd84-5rtgt"
Nov 01 05:09:08.376179 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:08.375877    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:09:08.652870 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:08.652839    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:09:08.711710 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:08.711679    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5f68b7bd84-5rtgt"]
Nov 01 05:09:08.716051 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:09:08.716021    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93b41328_171b_43ba_9a18_370d66e0cd5d.slice/crio-c8b30eac4dcf51c72c64f2a394c0b862fb9a3b05c56c7a65bdd8c3fd2b78c3b3 WatchSource:0}: Error finding container c8b30eac4dcf51c72c64f2a394c0b862fb9a3b05c56c7a65bdd8c3fd2b78c3b3: Status 404 returned error can't find the container with id c8b30eac4dcf51c72c64f2a394c0b862fb9a3b05c56c7a65bdd8c3fd2b78c3b3
Nov 01 05:09:08.717948 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:08.717930    5689 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 05:09:08.847696 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:08.847665    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:09:09.292530 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:09.292497    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5f68b7bd84-5rtgt" event={"ID":"93b41328-171b-43ba-9a18-370d66e0cd5d","Type":"ContainerStarted","Data":"c8b30eac4dcf51c72c64f2a394c0b862fb9a3b05c56c7a65bdd8c3fd2b78c3b3"}
Nov 01 05:09:09.305354 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:09.305327    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:09:09.477659 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:09.477622    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:09:09.906942 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:09.906907    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:09:10.132033 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:10.131972    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:09:12.308202 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:12.308169    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5f68b7bd84-5rtgt" event={"ID":"93b41328-171b-43ba-9a18-370d66e0cd5d","Type":"ContainerStarted","Data":"a060398becfc4b7f777a9b6fad563a34d44b8dc540a19877cf13914dc2d580c1"}
Nov 01 05:09:12.426163 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:12.426113    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5f68b7bd84-5rtgt" podStartSLOduration=2.617113194 podStartE2EDuration="5.426100885s" totalImagesPullingTime="2.808987691s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:09:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:09:12.425648948 +0000 UTC m=+330.496445670" watchObservedRunningTime="2026-11-01 05:09:12.426100885 +0000 UTC m=+330.496897601"
Nov 01 05:09:14.699101 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:14.699066    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:09:14.880219 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:14.880188    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:09:15.100965 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:15.100891    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:09:15.275692 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:15.275659    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:09:15.495016 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:15.494980    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:09:15.693205 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:15.693171    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:09:21.467557 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:21.467522    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:09:21.640448 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:21.640416    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:09:23.930756 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:23.930717    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:09:24.116580 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:24.116547    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:09:24.293536 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:24.293408    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:09:24.565427 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:24.565351    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:09:24.742959 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:24.742925    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:09:25.740547 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:25.740519    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-tzz7h"]
Nov 01 05:09:25.743745 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:25.743728    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-tzz7h"
Nov 01 05:09:25.795784 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:25.795752    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 01 05:09:25.795906 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:25.795873    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 01 05:09:25.845315 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:25.845290    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-gq664\""
Nov 01 05:09:25.846425 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:25.846406    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-tzz7h"]
Nov 01 05:09:25.893980 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:25.893954    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czf6k\" (UniqueName: \"kubernetes.io/projected/98cffadd-37d7-4cb8-8ff8-b79b00e803b7-kube-api-access-czf6k\") pod \"authorino-57f7f8659f-tzz7h\" (UID: \"98cffadd-37d7-4cb8-8ff8-b79b00e803b7\") " pod="kuadrant-system/authorino-57f7f8659f-tzz7h"
Nov 01 05:09:25.926368 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:25.926326    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:09:25.995181 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:25.995115    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czf6k\" (UniqueName: \"kubernetes.io/projected/98cffadd-37d7-4cb8-8ff8-b79b00e803b7-kube-api-access-czf6k\") pod \"authorino-57f7f8659f-tzz7h\" (UID: \"98cffadd-37d7-4cb8-8ff8-b79b00e803b7\") " pod="kuadrant-system/authorino-57f7f8659f-tzz7h"
Nov 01 05:09:26.056083 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:26.056051    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czf6k\" (UniqueName: \"kubernetes.io/projected/98cffadd-37d7-4cb8-8ff8-b79b00e803b7-kube-api-access-czf6k\") pod \"authorino-57f7f8659f-tzz7h\" (UID: \"98cffadd-37d7-4cb8-8ff8-b79b00e803b7\") " pod="kuadrant-system/authorino-57f7f8659f-tzz7h"
Nov 01 05:09:26.068262 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:26.068240    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-tzz7h"
Nov 01 05:09:26.104922 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:26.104886    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:09:26.435053 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:26.435021    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-tzz7h"]
Nov 01 05:09:26.439499 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:09:26.439419    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod98cffadd_37d7_4cb8_8ff8_b79b00e803b7.slice/crio-58171ee2d02022d589c8e02c72531d81d98495bd5d583e0130c9535ea98d0413 WatchSource:0}: Error finding container 58171ee2d02022d589c8e02c72531d81d98495bd5d583e0130c9535ea98d0413: Status 404 returned error can't find the container with id 58171ee2d02022d589c8e02c72531d81d98495bd5d583e0130c9535ea98d0413
Nov 01 05:09:27.368160 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:27.368128    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-tzz7h" event={"ID":"98cffadd-37d7-4cb8-8ff8-b79b00e803b7","Type":"ContainerStarted","Data":"58171ee2d02022d589c8e02c72531d81d98495bd5d583e0130c9535ea98d0413"}
Nov 01 05:09:28.451221 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:28.449753    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:09:28.453383 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:28.453347    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-dc89h"
Nov 01 05:09:28.505138 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:28.505107    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-ghcgj\""
Nov 01 05:09:28.520164 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:28.520137    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67498\" (UniqueName: \"kubernetes.io/projected/817d6a15-8f0e-406f-8dd2-bed3b6323796-kube-api-access-67498\") pod \"limitador-limitador-7c9b87d574-dc89h\" (UID: \"817d6a15-8f0e-406f-8dd2-bed3b6323796\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-dc89h"
Nov 01 05:09:28.520313 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:28.520190    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/817d6a15-8f0e-406f-8dd2-bed3b6323796-config-file\") pod \"limitador-limitador-7c9b87d574-dc89h\" (UID: \"817d6a15-8f0e-406f-8dd2-bed3b6323796\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-dc89h"
Nov 01 05:09:28.560820 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:28.554699    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Nov 01 05:09:28.560820 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:28.555791    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:09:28.621837 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:28.621803    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67498\" (UniqueName: \"kubernetes.io/projected/817d6a15-8f0e-406f-8dd2-bed3b6323796-kube-api-access-67498\") pod \"limitador-limitador-7c9b87d574-dc89h\" (UID: \"817d6a15-8f0e-406f-8dd2-bed3b6323796\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-dc89h"
Nov 01 05:09:28.621969 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:28.621856    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/817d6a15-8f0e-406f-8dd2-bed3b6323796-config-file\") pod \"limitador-limitador-7c9b87d574-dc89h\" (UID: \"817d6a15-8f0e-406f-8dd2-bed3b6323796\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-dc89h"
Nov 01 05:09:28.622565 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:28.622546    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/817d6a15-8f0e-406f-8dd2-bed3b6323796-config-file\") pod \"limitador-limitador-7c9b87d574-dc89h\" (UID: \"817d6a15-8f0e-406f-8dd2-bed3b6323796\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-dc89h"
Nov 01 05:09:28.685855 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:28.685828    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67498\" (UniqueName: \"kubernetes.io/projected/817d6a15-8f0e-406f-8dd2-bed3b6323796-kube-api-access-67498\") pod \"limitador-limitador-7c9b87d574-dc89h\" (UID: \"817d6a15-8f0e-406f-8dd2-bed3b6323796\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-dc89h"
Nov 01 05:09:28.754098 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:28.754021    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:09:28.771000 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:28.770814    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-dc89h"
Nov 01 05:09:29.163361 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:29.163330    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:09:29.167707 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:09:29.167673    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod817d6a15_8f0e_406f_8dd2_bed3b6323796.slice/crio-a3260bdde7ba737ab98fa03175f957bb85ae360e796cafc49a47dedf49a2482b WatchSource:0}: Error finding container a3260bdde7ba737ab98fa03175f957bb85ae360e796cafc49a47dedf49a2482b: Status 404 returned error can't find the container with id a3260bdde7ba737ab98fa03175f957bb85ae360e796cafc49a47dedf49a2482b
Nov 01 05:09:29.377829 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:29.377793    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-dc89h" event={"ID":"817d6a15-8f0e-406f-8dd2-bed3b6323796","Type":"ContainerStarted","Data":"a3260bdde7ba737ab98fa03175f957bb85ae360e796cafc49a47dedf49a2482b"}
Nov 01 05:09:29.468986 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:29.468895    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:09:29.692538 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:29.692503    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:09:30.069260 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:09:30.069222    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:09:30.383213 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:30.383128    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-tzz7h" event={"ID":"98cffadd-37d7-4cb8-8ff8-b79b00e803b7","Type":"ContainerStarted","Data":"409740adcc508d0bb5e7a24159bbdf418a7fa12ca31fabf55f20285ec593909d"}
Nov 01 05:09:31.388382 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:31.388347    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-dc89h" event={"ID":"817d6a15-8f0e-406f-8dd2-bed3b6323796","Type":"ContainerStarted","Data":"0b78d79b441aa7201df35d7577c4ef1bb75ba37ce48431f5bfc53aae5c4ceafe"}
Nov 01 05:09:31.388798 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:31.388566    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-dc89h"
Nov 01 05:09:31.505616 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:31.505558    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-dc89h" podStartSLOduration=1.814336093 podStartE2EDuration="3.505539399s" totalImagesPullingTime="1.691203306s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:09:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:09:31.505250137 +0000 UTC m=+349.576046884" watchObservedRunningTime="2026-11-01 05:09:31.505539399 +0000 UTC m=+349.576336120"
Nov 01 05:09:31.506000 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:31.505969    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-tzz7h" podStartSLOduration=2.835440749 podStartE2EDuration="6.50596124s" totalImagesPullingTime="3.670520491s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:09:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:09:30.498954111 +0000 UTC m=+348.569750830" watchObservedRunningTime="2026-11-01 05:09:31.50596124 +0000 UTC m=+349.576757959"
Nov 01 05:09:32.644937 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:32.644900    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-tzz7h"]
Nov 01 05:09:32.645402 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:32.645091    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-tzz7h" podUID="98cffadd-37d7-4cb8-8ff8-b79b00e803b7" containerName="authorino" containerID="cri-o://409740adcc508d0bb5e7a24159bbdf418a7fa12ca31fabf55f20285ec593909d" gracePeriod=30
Nov 01 05:09:32.893736 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:32.893712    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-tzz7h"
Nov 01 05:09:33.064604 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:33.064577    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98cffadd-37d7-4cb8-8ff8-b79b00e803b7-kube-api-access-czf6k\" (UniqueName: \"kubernetes.io/projected/98cffadd-37d7-4cb8-8ff8-b79b00e803b7-kube-api-access-czf6k\") pod \"98cffadd-37d7-4cb8-8ff8-b79b00e803b7\" (UID: \"98cffadd-37d7-4cb8-8ff8-b79b00e803b7\") "
Nov 01 05:09:33.066966 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:33.066939    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98cffadd-37d7-4cb8-8ff8-b79b00e803b7-kube-api-access-czf6k" pod "98cffadd-37d7-4cb8-8ff8-b79b00e803b7" (UID: "98cffadd-37d7-4cb8-8ff8-b79b00e803b7"). InnerVolumeSpecName "kube-api-access-czf6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:09:33.165051 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:33.165026    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czf6k\" (UniqueName: \"kubernetes.io/projected/98cffadd-37d7-4cb8-8ff8-b79b00e803b7-kube-api-access-czf6k\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:09:33.396625 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:33.396547    5689 generic.go:356] "Generic (PLEG): container finished" podID="98cffadd-37d7-4cb8-8ff8-b79b00e803b7" containerID="409740adcc508d0bb5e7a24159bbdf418a7fa12ca31fabf55f20285ec593909d" exitCode=0
Nov 01 05:09:33.396625 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:33.396589    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-tzz7h"
Nov 01 05:09:33.396786 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:33.396638    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-tzz7h" event={"ID":"98cffadd-37d7-4cb8-8ff8-b79b00e803b7","Type":"ContainerDied","Data":"409740adcc508d0bb5e7a24159bbdf418a7fa12ca31fabf55f20285ec593909d"}
Nov 01 05:09:33.396786 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:33.396677    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-tzz7h" event={"ID":"98cffadd-37d7-4cb8-8ff8-b79b00e803b7","Type":"ContainerDied","Data":"58171ee2d02022d589c8e02c72531d81d98495bd5d583e0130c9535ea98d0413"}
Nov 01 05:09:33.396786 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:33.396694    5689 scope.go:122] "RemoveContainer" containerID="409740adcc508d0bb5e7a24159bbdf418a7fa12ca31fabf55f20285ec593909d"
Nov 01 05:09:33.405203 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:33.405186    5689 scope.go:122] "RemoveContainer" containerID="409740adcc508d0bb5e7a24159bbdf418a7fa12ca31fabf55f20285ec593909d"
Nov 01 05:09:33.405564 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:09:33.405545    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"409740adcc508d0bb5e7a24159bbdf418a7fa12ca31fabf55f20285ec593909d\": container with ID starting with 409740adcc508d0bb5e7a24159bbdf418a7fa12ca31fabf55f20285ec593909d not found: ID does not exist" containerID="409740adcc508d0bb5e7a24159bbdf418a7fa12ca31fabf55f20285ec593909d"
Nov 01 05:09:33.405636 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:33.405573    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"409740adcc508d0bb5e7a24159bbdf418a7fa12ca31fabf55f20285ec593909d"} err="failed to get container status \"409740adcc508d0bb5e7a24159bbdf418a7fa12ca31fabf55f20285ec593909d\": rpc error: code = NotFound desc = could not find container \"409740adcc508d0bb5e7a24159bbdf418a7fa12ca31fabf55f20285ec593909d\": container with ID starting with 409740adcc508d0bb5e7a24159bbdf418a7fa12ca31fabf55f20285ec593909d not found: ID does not exist"
Nov 01 05:09:33.539469 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:33.539429    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:09:33.566657 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:33.566626    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-tzz7h"]
Nov 01 05:09:33.567224 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:33.567207    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-tzz7h"]
Nov 01 05:09:33.715874 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:33.715833    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:09:34.066223 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:34.066142    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98cffadd-37d7-4cb8-8ff8-b79b00e803b7" path="/var/lib/kubelet/pods/98cffadd-37d7-4cb8-8ff8-b79b00e803b7/volumes"
Nov 01 05:09:40.579029 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:40.578978    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:09:40.755168 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:40.755124    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:09:42.392359 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:42.392288    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-dc89h"
Nov 01 05:09:43.057025 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:43.056991    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:09:43.242438 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:43.242407    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:09:46.227737 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:46.227700    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:09:46.432033 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:46.432002    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:09:46.609640 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:46.609563    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:09:47.770817 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:47.770785    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:09:47.958418 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:47.958385    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:09:52.852510 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:52.852413    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:09:53.024594 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:53.024559    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:09:53.207201 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:53.207160    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:09:53.385705 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:53.385638    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:09:53.592302 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:53.592203    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:09:53.760721 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:53.760683    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:09:53.991369 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:53.991337    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:09:54.172644 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:54.172613    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:09:54.424608 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:54.424580    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:09:55.135801 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:55.135768    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-g999k"]
Nov 01 05:09:55.136266 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:55.136249    5689 cpu_manager.go:414] "containerMap: removing container" podUID="98cffadd-37d7-4cb8-8ff8-b79b00e803b7" containerName="authorino"
Nov 01 05:09:55.136331 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:55.136273    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98cffadd-37d7-4cb8-8ff8-b79b00e803b7" containerName="authorino"
Nov 01 05:09:55.136385 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:55.136370    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="98cffadd-37d7-4cb8-8ff8-b79b00e803b7" containerName="authorino"
Nov 01 05:09:55.139468 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:55.139427    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-g999k"
Nov 01 05:09:55.191936 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:55.191912    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 05:09:55.191936 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:55.191912    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 05:09:55.236548 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:55.236522    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c552fda2-a502-4a75-94cc-dcdf0b7087fb-data\") pod \"postgres-7cdbf6dc44-g999k\" (UID: \"c552fda2-a502-4a75-94cc-dcdf0b7087fb\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-g999k"
Nov 01 05:09:55.236674 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:55.236572    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r979j\" (UniqueName: \"kubernetes.io/projected/c552fda2-a502-4a75-94cc-dcdf0b7087fb-kube-api-access-r979j\") pod \"postgres-7cdbf6dc44-g999k\" (UID: \"c552fda2-a502-4a75-94cc-dcdf0b7087fb\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-g999k"
Nov 01 05:09:55.240216 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:55.240180    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:09:55.241418 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:55.241395    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Nov 01 05:09:55.241628 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:55.241522    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-vrbt2\""
Nov 01 05:09:55.242399 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:55.242381    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-g999k"]
Nov 01 05:09:55.337509 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:55.337440    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c552fda2-a502-4a75-94cc-dcdf0b7087fb-data\") pod \"postgres-7cdbf6dc44-g999k\" (UID: \"c552fda2-a502-4a75-94cc-dcdf0b7087fb\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-g999k"
Nov 01 05:09:55.337651 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:55.337547    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r979j\" (UniqueName: \"kubernetes.io/projected/c552fda2-a502-4a75-94cc-dcdf0b7087fb-kube-api-access-r979j\") pod \"postgres-7cdbf6dc44-g999k\" (UID: \"c552fda2-a502-4a75-94cc-dcdf0b7087fb\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-g999k"
Nov 01 05:09:55.337831 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:55.337814    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c552fda2-a502-4a75-94cc-dcdf0b7087fb-data\") pod \"postgres-7cdbf6dc44-g999k\" (UID: \"c552fda2-a502-4a75-94cc-dcdf0b7087fb\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-g999k"
Nov 01 05:09:55.398667 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:55.398639    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r979j\" (UniqueName: \"kubernetes.io/projected/c552fda2-a502-4a75-94cc-dcdf0b7087fb-kube-api-access-r979j\") pod \"postgres-7cdbf6dc44-g999k\" (UID: \"c552fda2-a502-4a75-94cc-dcdf0b7087fb\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-g999k"
Nov 01 05:09:55.453534 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:55.453510    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-g999k"
Nov 01 05:09:55.494425 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:55.494399    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:09:55.823557 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:55.823481    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-g999k"]
Nov 01 05:09:55.826921 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:09:55.826894    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc552fda2_a502_4a75_94cc_dcdf0b7087fb.slice/crio-1ec47d66ea46c8087cf879e5002a5ccf58b57c446296fba4216179ffe982e9ac WatchSource:0}: Error finding container 1ec47d66ea46c8087cf879e5002a5ccf58b57c446296fba4216179ffe982e9ac: Status 404 returned error can't find the container with id 1ec47d66ea46c8087cf879e5002a5ccf58b57c446296fba4216179ffe982e9ac
Nov 01 05:09:56.489012 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:09:56.488933    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-g999k" event={"ID":"c552fda2-a502-4a75-94cc-dcdf0b7087fb","Type":"ContainerStarted","Data":"1ec47d66ea46c8087cf879e5002a5ccf58b57c446296fba4216179ffe982e9ac"}
Nov 01 05:10:00.074433 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:10:00.074391    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:10:00.387723 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:00.387640    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:10:00.557870 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:00.557834    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:10:02.515185 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:02.515153    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-g999k" event={"ID":"c552fda2-a502-4a75-94cc-dcdf0b7087fb","Type":"ContainerStarted","Data":"b6b622d28224906ef580c4cea150c1a3ed0720eeec5884972841c12272f05283"}
Nov 01 05:10:02.515660 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:02.515273    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-g999k"
Nov 01 05:10:02.630006 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:02.629945    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-g999k" podStartSLOduration=1.310513668 podStartE2EDuration="7.629930656s" totalImagesPullingTime="6.319416988s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:09:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:10:02.629406051 +0000 UTC m=+380.700202771" watchObservedRunningTime="2026-11-01 05:10:02.629930656 +0000 UTC m=+380.700727377"
Nov 01 05:10:02.853722 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:02.853636    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:10:03.031839 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:03.031805    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:10:08.550574 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:08.550545    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-g999k"
Nov 01 05:10:15.606070 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:15.606034    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:10:15.789290 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:15.789257    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:10:15.961658 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:15.961623    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:10:16.146088 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:16.146053    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:10:16.950194 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:16.950153    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:10:17.139235 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:17.139204    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:10:23.321935 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:23.321898    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:10:23.507652 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:23.507617    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:10:26.502709 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:26.502608    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:10:26.685080 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:26.685035    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:10:30.067161 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:10:30.067133    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:10:37.897970 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:37.897936    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zpxqq"]
Nov 01 05:10:37.912245 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:37.912220    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zpxqq"
Nov 01 05:10:37.914121 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:37.914098    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b54edd8b-ea2a-4035-9dd6-c4628eb73209-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zpxqq\" (UID: \"b54edd8b-ea2a-4035-9dd6-c4628eb73209\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zpxqq"
Nov 01 05:10:37.914252 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:37.914128    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b54edd8b-ea2a-4035-9dd6-c4628eb73209-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zpxqq\" (UID: \"b54edd8b-ea2a-4035-9dd6-c4628eb73209\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zpxqq"
Nov 01 05:10:37.914252 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:37.914161    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b54edd8b-ea2a-4035-9dd6-c4628eb73209-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zpxqq\" (UID: \"b54edd8b-ea2a-4035-9dd6-c4628eb73209\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zpxqq"
Nov 01 05:10:37.914342 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:37.914322    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b54edd8b-ea2a-4035-9dd6-c4628eb73209-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zpxqq\" (UID: \"b54edd8b-ea2a-4035-9dd6-c4628eb73209\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zpxqq"
Nov 01 05:10:37.914377 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:37.914363    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b54edd8b-ea2a-4035-9dd6-c4628eb73209-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zpxqq\" (UID: \"b54edd8b-ea2a-4035-9dd6-c4628eb73209\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zpxqq"
Nov 01 05:10:37.914410 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:37.914393    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzkr6\" (UniqueName: \"kubernetes.io/projected/b54edd8b-ea2a-4035-9dd6-c4628eb73209-kube-api-access-wzkr6\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zpxqq\" (UID: \"b54edd8b-ea2a-4035-9dd6-c4628eb73209\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zpxqq"
Nov 01 05:10:37.914503 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:37.914479    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b54edd8b-ea2a-4035-9dd6-c4628eb73209-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zpxqq\" (UID: \"b54edd8b-ea2a-4035-9dd6-c4628eb73209\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zpxqq"
Nov 01 05:10:38.015035 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:38.015006    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zpxqq"]
Nov 01 05:10:38.015277 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:38.015257    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzkr6\" (UniqueName: \"kubernetes.io/projected/b54edd8b-ea2a-4035-9dd6-c4628eb73209-kube-api-access-wzkr6\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zpxqq\" (UID: \"b54edd8b-ea2a-4035-9dd6-c4628eb73209\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zpxqq"
Nov 01 05:10:38.015319 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:38.015293    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b54edd8b-ea2a-4035-9dd6-c4628eb73209-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zpxqq\" (UID: \"b54edd8b-ea2a-4035-9dd6-c4628eb73209\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zpxqq"
Nov 01 05:10:38.015353 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:38.015318    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b54edd8b-ea2a-4035-9dd6-c4628eb73209-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zpxqq\" (UID: \"b54edd8b-ea2a-4035-9dd6-c4628eb73209\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zpxqq"
Nov 01 05:10:38.015353 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:38.015342    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b54edd8b-ea2a-4035-9dd6-c4628eb73209-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zpxqq\" (UID: \"b54edd8b-ea2a-4035-9dd6-c4628eb73209\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zpxqq"
Nov 01 05:10:38.015417 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:38.015385    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b54edd8b-ea2a-4035-9dd6-c4628eb73209-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zpxqq\" (UID: \"b54edd8b-ea2a-4035-9dd6-c4628eb73209\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zpxqq"
Nov 01 05:10:38.015469 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:38.015421    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b54edd8b-ea2a-4035-9dd6-c4628eb73209-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zpxqq\" (UID: \"b54edd8b-ea2a-4035-9dd6-c4628eb73209\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zpxqq"
Nov 01 05:10:38.015514 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:38.015479    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b54edd8b-ea2a-4035-9dd6-c4628eb73209-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zpxqq\" (UID: \"b54edd8b-ea2a-4035-9dd6-c4628eb73209\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zpxqq"
Nov 01 05:10:38.016419 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:38.016391    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b54edd8b-ea2a-4035-9dd6-c4628eb73209-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zpxqq\" (UID: \"b54edd8b-ea2a-4035-9dd6-c4628eb73209\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zpxqq"
Nov 01 05:10:38.017887 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:38.017859    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b54edd8b-ea2a-4035-9dd6-c4628eb73209-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zpxqq\" (UID: \"b54edd8b-ea2a-4035-9dd6-c4628eb73209\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zpxqq"
Nov 01 05:10:38.018005 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:38.017924    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b54edd8b-ea2a-4035-9dd6-c4628eb73209-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zpxqq\" (UID: \"b54edd8b-ea2a-4035-9dd6-c4628eb73209\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zpxqq"
Nov 01 05:10:38.018061 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:38.017994    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b54edd8b-ea2a-4035-9dd6-c4628eb73209-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zpxqq\" (UID: \"b54edd8b-ea2a-4035-9dd6-c4628eb73209\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zpxqq"
Nov 01 05:10:38.018181 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:38.018165    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b54edd8b-ea2a-4035-9dd6-c4628eb73209-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zpxqq\" (UID: \"b54edd8b-ea2a-4035-9dd6-c4628eb73209\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zpxqq"
Nov 01 05:10:38.076074 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:38.076042    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b54edd8b-ea2a-4035-9dd6-c4628eb73209-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zpxqq\" (UID: \"b54edd8b-ea2a-4035-9dd6-c4628eb73209\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zpxqq"
Nov 01 05:10:38.078274 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:38.078249    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzkr6\" (UniqueName: \"kubernetes.io/projected/b54edd8b-ea2a-4035-9dd6-c4628eb73209-kube-api-access-wzkr6\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zpxqq\" (UID: \"b54edd8b-ea2a-4035-9dd6-c4628eb73209\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zpxqq"
Nov 01 05:10:38.223409 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:38.223342    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zpxqq"
Nov 01 05:10:38.664281 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:38.664245    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zpxqq"]
Nov 01 05:10:38.667019 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:10:38.666991    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb54edd8b_ea2a_4035_9dd6_c4628eb73209.slice/crio-01a9661cb02bd203a64e3b484850a0ace3dad40fefff44ee19d85cdaa5e8ca72 WatchSource:0}: Error finding container 01a9661cb02bd203a64e3b484850a0ace3dad40fefff44ee19d85cdaa5e8ca72: Status 404 returned error can't find the container with id 01a9661cb02bd203a64e3b484850a0ace3dad40fefff44ee19d85cdaa5e8ca72
Nov 01 05:10:38.669037 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:38.669007    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:10:38.669108 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:38.669065    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:10:39.666345 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:39.666311    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zpxqq" event={"ID":"b54edd8b-ea2a-4035-9dd6-c4628eb73209","Type":"ContainerStarted","Data":"8c599932ea4bb161ce0617aeb7703190520d777ff4555c5fb0ee810a3d426abb"}
Nov 01 05:10:39.666776 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:39.666361    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zpxqq" event={"ID":"b54edd8b-ea2a-4035-9dd6-c4628eb73209","Type":"ContainerStarted","Data":"01a9661cb02bd203a64e3b484850a0ace3dad40fefff44ee19d85cdaa5e8ca72"}
Nov 01 05:10:39.666776 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:39.666482    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zpxqq"
Nov 01 05:10:39.667912 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:39.667878    5689 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-zpxqq container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 05:10:39.668005 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:39.667934    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zpxqq" podUID="b54edd8b-ea2a-4035-9dd6-c4628eb73209" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 05:10:39.870935 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:39.870872    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zpxqq" podStartSLOduration=2.870855904 podStartE2EDuration="2.870855904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:10:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:10:39.835972557 +0000 UTC m=+417.906769299" watchObservedRunningTime="2026-11-01 05:10:39.870855904 +0000 UTC m=+417.941652628"
Nov 01 05:10:40.670523 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:40.670497    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zpxqq"
Nov 01 05:10:40.890777 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:40.890743    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-8f57g"]
Nov 01 05:10:40.891050 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:40.891001    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8f57g" podUID="2a0b9b72-9a54-4977-87eb-7a341c1156ea" containerName="discovery" containerID="cri-o://44ccba72ba73cad0027a6892995c415b8a1db65b0144ffb482014f1d5061349a" gracePeriod=30
Nov 01 05:10:41.137990 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:41.137968    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8f57g"
Nov 01 05:10:41.240530 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:41.240442    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a0b9b72-9a54-4977-87eb-7a341c1156ea-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2a0b9b72-9a54-4977-87eb-7a341c1156ea-istio-csr-ca-configmap\") pod \"2a0b9b72-9a54-4977-87eb-7a341c1156ea\" (UID: \"2a0b9b72-9a54-4977-87eb-7a341c1156ea\") "
Nov 01 05:10:41.240530 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:41.240503    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a0b9b72-9a54-4977-87eb-7a341c1156ea-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2a0b9b72-9a54-4977-87eb-7a341c1156ea-istio-kubeconfig\") pod \"2a0b9b72-9a54-4977-87eb-7a341c1156ea\" (UID: \"2a0b9b72-9a54-4977-87eb-7a341c1156ea\") "
Nov 01 05:10:41.240530 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:41.240521    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a0b9b72-9a54-4977-87eb-7a341c1156ea-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2a0b9b72-9a54-4977-87eb-7a341c1156ea-istio-csr-dns-cert\") pod \"2a0b9b72-9a54-4977-87eb-7a341c1156ea\" (UID: \"2a0b9b72-9a54-4977-87eb-7a341c1156ea\") "
Nov 01 05:10:41.240738 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:41.240556    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a0b9b72-9a54-4977-87eb-7a341c1156ea-kube-api-access-2cq6f\" (UniqueName: \"kubernetes.io/projected/2a0b9b72-9a54-4977-87eb-7a341c1156ea-kube-api-access-2cq6f\") pod \"2a0b9b72-9a54-4977-87eb-7a341c1156ea\" (UID: \"2a0b9b72-9a54-4977-87eb-7a341c1156ea\") "
Nov 01 05:10:41.240738 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:41.240588    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a0b9b72-9a54-4977-87eb-7a341c1156ea-istio-token\" (UniqueName: \"kubernetes.io/projected/2a0b9b72-9a54-4977-87eb-7a341c1156ea-istio-token\") pod \"2a0b9b72-9a54-4977-87eb-7a341c1156ea\" (UID: \"2a0b9b72-9a54-4977-87eb-7a341c1156ea\") "
Nov 01 05:10:41.240840 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:41.240745    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a0b9b72-9a54-4977-87eb-7a341c1156ea-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a0b9b72-9a54-4977-87eb-7a341c1156ea-local-certs\") pod \"2a0b9b72-9a54-4977-87eb-7a341c1156ea\" (UID: \"2a0b9b72-9a54-4977-87eb-7a341c1156ea\") "
Nov 01 05:10:41.240840 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:41.240813    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a0b9b72-9a54-4977-87eb-7a341c1156ea-cacerts\" (UniqueName: \"kubernetes.io/secret/2a0b9b72-9a54-4977-87eb-7a341c1156ea-cacerts\") pod \"2a0b9b72-9a54-4977-87eb-7a341c1156ea\" (UID: \"2a0b9b72-9a54-4977-87eb-7a341c1156ea\") "
Nov 01 05:10:41.240934 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:41.240861    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a0b9b72-9a54-4977-87eb-7a341c1156ea-istio-csr-ca-configmap" pod "2a0b9b72-9a54-4977-87eb-7a341c1156ea" (UID: "2a0b9b72-9a54-4977-87eb-7a341c1156ea"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:10:41.241104 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:41.241083    5689 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2a0b9b72-9a54-4977-87eb-7a341c1156ea-istio-csr-ca-configmap\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:10:41.243097 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:41.243072    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a0b9b72-9a54-4977-87eb-7a341c1156ea-kube-api-access-2cq6f" pod "2a0b9b72-9a54-4977-87eb-7a341c1156ea" (UID: "2a0b9b72-9a54-4977-87eb-7a341c1156ea"). InnerVolumeSpecName "kube-api-access-2cq6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:10:41.243224 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:41.243208    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a0b9b72-9a54-4977-87eb-7a341c1156ea-istio-csr-dns-cert" pod "2a0b9b72-9a54-4977-87eb-7a341c1156ea" (UID: "2a0b9b72-9a54-4977-87eb-7a341c1156ea"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 05:10:41.243340 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:41.243321    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a0b9b72-9a54-4977-87eb-7a341c1156ea-istio-token" pod "2a0b9b72-9a54-4977-87eb-7a341c1156ea" (UID: "2a0b9b72-9a54-4977-87eb-7a341c1156ea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:10:41.243389 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:41.243354    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a0b9b72-9a54-4977-87eb-7a341c1156ea-istio-kubeconfig" pod "2a0b9b72-9a54-4977-87eb-7a341c1156ea" (UID: "2a0b9b72-9a54-4977-87eb-7a341c1156ea"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 05:10:41.243422 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:41.243382    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a0b9b72-9a54-4977-87eb-7a341c1156ea-cacerts" pod "2a0b9b72-9a54-4977-87eb-7a341c1156ea" (UID: "2a0b9b72-9a54-4977-87eb-7a341c1156ea"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 05:10:41.243579 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:41.243554    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a0b9b72-9a54-4977-87eb-7a341c1156ea-local-certs" pod "2a0b9b72-9a54-4977-87eb-7a341c1156ea" (UID: "2a0b9b72-9a54-4977-87eb-7a341c1156ea"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:10:41.342516 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:41.342482    5689 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2a0b9b72-9a54-4977-87eb-7a341c1156ea-cacerts\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:10:41.342516 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:41.342516    5689 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2a0b9b72-9a54-4977-87eb-7a341c1156ea-istio-kubeconfig\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:10:41.342701 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:41.342529    5689 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2a0b9b72-9a54-4977-87eb-7a341c1156ea-istio-csr-dns-cert\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:10:41.342701 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:41.342538    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cq6f\" (UniqueName: \"kubernetes.io/projected/2a0b9b72-9a54-4977-87eb-7a341c1156ea-kube-api-access-2cq6f\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:10:41.342701 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:41.342549    5689 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a0b9b72-9a54-4977-87eb-7a341c1156ea-istio-token\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:10:41.342701 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:41.342557    5689 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a0b9b72-9a54-4977-87eb-7a341c1156ea-local-certs\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:10:41.675148 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:41.675109    5689 generic.go:356] "Generic (PLEG): container finished" podID="2a0b9b72-9a54-4977-87eb-7a341c1156ea" containerID="44ccba72ba73cad0027a6892995c415b8a1db65b0144ffb482014f1d5061349a" exitCode=0
Nov 01 05:10:41.675586 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:41.675154    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8f57g"
Nov 01 05:10:41.675586 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:41.675201    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8f57g" event={"ID":"2a0b9b72-9a54-4977-87eb-7a341c1156ea","Type":"ContainerDied","Data":"44ccba72ba73cad0027a6892995c415b8a1db65b0144ffb482014f1d5061349a"}
Nov 01 05:10:41.675586 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:41.675241    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8f57g" event={"ID":"2a0b9b72-9a54-4977-87eb-7a341c1156ea","Type":"ContainerDied","Data":"c7efe99a65ea7e18a2af48691ea94ca0242a88b53a344f5481d558032f696979"}
Nov 01 05:10:41.675586 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:41.675266    5689 scope.go:122] "RemoveContainer" containerID="44ccba72ba73cad0027a6892995c415b8a1db65b0144ffb482014f1d5061349a"
Nov 01 05:10:41.683909 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:41.683895    5689 scope.go:122] "RemoveContainer" containerID="44ccba72ba73cad0027a6892995c415b8a1db65b0144ffb482014f1d5061349a"
Nov 01 05:10:41.684156 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:10:41.684138    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44ccba72ba73cad0027a6892995c415b8a1db65b0144ffb482014f1d5061349a\": container with ID starting with 44ccba72ba73cad0027a6892995c415b8a1db65b0144ffb482014f1d5061349a not found: ID does not exist" containerID="44ccba72ba73cad0027a6892995c415b8a1db65b0144ffb482014f1d5061349a"
Nov 01 05:10:41.684223 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:41.684168    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44ccba72ba73cad0027a6892995c415b8a1db65b0144ffb482014f1d5061349a"} err="failed to get container status \"44ccba72ba73cad0027a6892995c415b8a1db65b0144ffb482014f1d5061349a\": rpc error: code = NotFound desc = could not find container \"44ccba72ba73cad0027a6892995c415b8a1db65b0144ffb482014f1d5061349a\": container with ID starting with 44ccba72ba73cad0027a6892995c415b8a1db65b0144ffb482014f1d5061349a not found: ID does not exist"
Nov 01 05:10:41.839657 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:41.839627    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-8f57g"]
Nov 01 05:10:41.889071 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:41.889050    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-8f57g"]
Nov 01 05:10:42.066513 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:42.066422    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a0b9b72-9a54-4977-87eb-7a341c1156ea" path="/var/lib/kubelet/pods/2a0b9b72-9a54-4977-87eb-7a341c1156ea/volumes"
Nov 01 05:10:42.651042 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:42.651012    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:10:42.823791 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:42.823761    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:10:43.002393 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:43.002309    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:10:43.178569 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:43.178532    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:10:43.397029 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:43.397002    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:10:43.528223 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:43.528189    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-hstph"]
Nov 01 05:10:43.528635 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:43.528615    5689 cpu_manager.go:414] "containerMap: removing container" podUID="2a0b9b72-9a54-4977-87eb-7a341c1156ea" containerName="discovery"
Nov 01 05:10:43.528635 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:43.528635    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a0b9b72-9a54-4977-87eb-7a341c1156ea" containerName="discovery"
Nov 01 05:10:43.528793 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:43.528711    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a0b9b72-9a54-4977-87eb-7a341c1156ea" containerName="discovery"
Nov 01 05:10:43.532880 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:43.532860    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-hstph"
Nov 01 05:10:43.560861 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:43.560838    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4959df17-a373-4cef-b555-e342fb4ebbdf-cert\") pod \"kserve-controller-manager-776cc66d47-hstph\" (UID: \"4959df17-a373-4cef-b555-e342fb4ebbdf\") " pod="opendatahub/kserve-controller-manager-776cc66d47-hstph"
Nov 01 05:10:43.560964 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:43.560945    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9m4f\" (UniqueName: \"kubernetes.io/projected/4959df17-a373-4cef-b555-e342fb4ebbdf-kube-api-access-v9m4f\") pod \"kserve-controller-manager-776cc66d47-hstph\" (UID: \"4959df17-a373-4cef-b555-e342fb4ebbdf\") " pod="opendatahub/kserve-controller-manager-776cc66d47-hstph"
Nov 01 05:10:43.581888 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:43.581861    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:10:43.585081 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:43.585059    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-smgzb\""
Nov 01 05:10:43.585138 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:43.585100    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Nov 01 05:10:43.585138 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:43.585123    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 01 05:10:43.585138 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:43.585135    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 01 05:10:43.590709 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:43.590685    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-hstph"]
Nov 01 05:10:43.661856 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:43.661790    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9m4f\" (UniqueName: \"kubernetes.io/projected/4959df17-a373-4cef-b555-e342fb4ebbdf-kube-api-access-v9m4f\") pod \"kserve-controller-manager-776cc66d47-hstph\" (UID: \"4959df17-a373-4cef-b555-e342fb4ebbdf\") " pod="opendatahub/kserve-controller-manager-776cc66d47-hstph"
Nov 01 05:10:43.661856 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:43.661828    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4959df17-a373-4cef-b555-e342fb4ebbdf-cert\") pod \"kserve-controller-manager-776cc66d47-hstph\" (UID: \"4959df17-a373-4cef-b555-e342fb4ebbdf\") " pod="opendatahub/kserve-controller-manager-776cc66d47-hstph"
Nov 01 05:10:43.664297 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:43.664272    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4959df17-a373-4cef-b555-e342fb4ebbdf-cert\") pod \"kserve-controller-manager-776cc66d47-hstph\" (UID: \"4959df17-a373-4cef-b555-e342fb4ebbdf\") " pod="opendatahub/kserve-controller-manager-776cc66d47-hstph"
Nov 01 05:10:43.722269 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:43.722238    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9m4f\" (UniqueName: \"kubernetes.io/projected/4959df17-a373-4cef-b555-e342fb4ebbdf-kube-api-access-v9m4f\") pod \"kserve-controller-manager-776cc66d47-hstph\" (UID: \"4959df17-a373-4cef-b555-e342fb4ebbdf\") " pod="opendatahub/kserve-controller-manager-776cc66d47-hstph"
Nov 01 05:10:43.784796 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:43.784773    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:10:43.844358 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:43.844332    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-hstph"
Nov 01 05:10:43.864310 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:43.864225    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-pjms7"]
Nov 01 05:10:43.867749 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:43.867727    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-pjms7"
Nov 01 05:10:43.919978 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:43.919907    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Nov 01 05:10:43.919978 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:43.919916    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Nov 01 05:10:43.919978 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:43.919916    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-tjxrt\""
Nov 01 05:10:43.926728 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:43.926699    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-pjms7"]
Nov 01 05:10:43.964911 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:43.964884    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ff4e020b-0316-4b29-a761-904818e3fb4f-cert\") pod \"odh-model-controller-769875db56-pjms7\" (UID: \"ff4e020b-0316-4b29-a761-904818e3fb4f\") " pod="opendatahub/odh-model-controller-769875db56-pjms7"
Nov 01 05:10:43.965023 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:43.964920    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbs2d\" (UniqueName: \"kubernetes.io/projected/ff4e020b-0316-4b29-a761-904818e3fb4f-kube-api-access-jbs2d\") pod \"odh-model-controller-769875db56-pjms7\" (UID: \"ff4e020b-0316-4b29-a761-904818e3fb4f\") " pod="opendatahub/odh-model-controller-769875db56-pjms7"
Nov 01 05:10:43.965064 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:43.965030    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff4e020b-0316-4b29-a761-904818e3fb4f-metrics-certs\") pod \"odh-model-controller-769875db56-pjms7\" (UID: \"ff4e020b-0316-4b29-a761-904818e3fb4f\") " pod="opendatahub/odh-model-controller-769875db56-pjms7"
Nov 01 05:10:44.065486 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:44.065437    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ff4e020b-0316-4b29-a761-904818e3fb4f-cert\") pod \"odh-model-controller-769875db56-pjms7\" (UID: \"ff4e020b-0316-4b29-a761-904818e3fb4f\") " pod="opendatahub/odh-model-controller-769875db56-pjms7"
Nov 01 05:10:44.065626 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:44.065501    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbs2d\" (UniqueName: \"kubernetes.io/projected/ff4e020b-0316-4b29-a761-904818e3fb4f-kube-api-access-jbs2d\") pod \"odh-model-controller-769875db56-pjms7\" (UID: \"ff4e020b-0316-4b29-a761-904818e3fb4f\") " pod="opendatahub/odh-model-controller-769875db56-pjms7"
Nov 01 05:10:44.065626 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:44.065578    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff4e020b-0316-4b29-a761-904818e3fb4f-metrics-certs\") pod \"odh-model-controller-769875db56-pjms7\" (UID: \"ff4e020b-0316-4b29-a761-904818e3fb4f\") " pod="opendatahub/odh-model-controller-769875db56-pjms7"
Nov 01 05:10:44.068227 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:44.068203    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff4e020b-0316-4b29-a761-904818e3fb4f-metrics-certs\") pod \"odh-model-controller-769875db56-pjms7\" (UID: \"ff4e020b-0316-4b29-a761-904818e3fb4f\") " pod="opendatahub/odh-model-controller-769875db56-pjms7"
Nov 01 05:10:44.068335 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:44.068203    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ff4e020b-0316-4b29-a761-904818e3fb4f-cert\") pod \"odh-model-controller-769875db56-pjms7\" (UID: \"ff4e020b-0316-4b29-a761-904818e3fb4f\") " pod="opendatahub/odh-model-controller-769875db56-pjms7"
Nov 01 05:10:44.125911 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:44.125891    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbs2d\" (UniqueName: \"kubernetes.io/projected/ff4e020b-0316-4b29-a761-904818e3fb4f-kube-api-access-jbs2d\") pod \"odh-model-controller-769875db56-pjms7\" (UID: \"ff4e020b-0316-4b29-a761-904818e3fb4f\") " pod="opendatahub/odh-model-controller-769875db56-pjms7"
Nov 01 05:10:44.184761 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:44.184703    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-pjms7"
Nov 01 05:10:44.220261 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:44.220221    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-hstph"]
Nov 01 05:10:44.224683 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:10:44.224651    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4959df17_a373_4cef_b555_e342fb4ebbdf.slice/crio-d3458c369d1083ce8eb0ac918bdc08b19835d9acc5c6a4e310b51115f40f926c WatchSource:0}: Error finding container d3458c369d1083ce8eb0ac918bdc08b19835d9acc5c6a4e310b51115f40f926c: Status 404 returned error can't find the container with id d3458c369d1083ce8eb0ac918bdc08b19835d9acc5c6a4e310b51115f40f926c
Nov 01 05:10:44.554993 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:44.554919    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-pjms7"]
Nov 01 05:10:44.558942 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:10:44.558916    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff4e020b_0316_4b29_a761_904818e3fb4f.slice/crio-5712e816393e14200490b4f35a662877fabd4b1bb87602977afcd7e8f4c2e63d WatchSource:0}: Error finding container 5712e816393e14200490b4f35a662877fabd4b1bb87602977afcd7e8f4c2e63d: Status 404 returned error can't find the container with id 5712e816393e14200490b4f35a662877fabd4b1bb87602977afcd7e8f4c2e63d
Nov 01 05:10:44.688912 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:44.688880    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-hstph" event={"ID":"4959df17-a373-4cef-b555-e342fb4ebbdf","Type":"ContainerStarted","Data":"d3458c369d1083ce8eb0ac918bdc08b19835d9acc5c6a4e310b51115f40f926c"}
Nov 01 05:10:44.690003 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:44.689978    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-pjms7" event={"ID":"ff4e020b-0316-4b29-a761-904818e3fb4f","Type":"ContainerStarted","Data":"5712e816393e14200490b4f35a662877fabd4b1bb87602977afcd7e8f4c2e63d"}
Nov 01 05:10:44.924283 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:44.924255    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:10:47.706555 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:47.706519    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-hstph" event={"ID":"4959df17-a373-4cef-b555-e342fb4ebbdf","Type":"ContainerStarted","Data":"455057bbacbeed2b8d1100b8b074e1462875c0cb95d97c8c9f5131a779e33210"}
Nov 01 05:10:47.707027 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:47.706660    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-hstph"
Nov 01 05:10:47.824692 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:47.824422    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-hstph" podStartSLOduration=2.042981595 podStartE2EDuration="4.824403559s" totalImagesPullingTime="2.781421964s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:10:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:10:47.823073145 +0000 UTC m=+425.893869865" watchObservedRunningTime="2026-11-01 05:10:47.824403559 +0000 UTC m=+425.895200280"
Nov 01 05:10:48.255213 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:48.255181    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:10:49.716377 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:49.716345    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-pjms7" event={"ID":"ff4e020b-0316-4b29-a761-904818e3fb4f","Type":"ContainerStarted","Data":"b5eac5e8b2d85a5bda032dc3343864cfdf078e1dbcaa21671447e7343398236f"}
Nov 01 05:10:49.716770 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:49.716476    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-pjms7"
Nov 01 05:10:49.832682 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:49.832629    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-pjms7" podStartSLOduration=2.4645581610000002 podStartE2EDuration="6.832613501s" totalImagesPullingTime="4.36805534s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:10:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:10:49.831958847 +0000 UTC m=+427.902755602" watchObservedRunningTime="2026-11-01 05:10:49.832613501 +0000 UTC m=+427.903410298"
Nov 01 05:10:50.556038 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:50.556001    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:10:50.734237 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:50.734207    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:10:50.913791 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:50.913747    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:10:51.089910 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:51.089878    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:10:51.268759 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:51.268686    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:10:51.437329 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:51.437301    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:10:51.613139 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:51.613063    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:10:51.797357 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:51.797317    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:10:53.171799 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:53.171764    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:10:53.362155 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:53.362122    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:10:55.255830 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:55.255795    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:10:55.432727 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:55.432676    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:10:55.788850 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:55.788818    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:10:55.959945 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:55.959899    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:10:56.307833 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:56.307792    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7688455878-p5cqw"]
Nov 01 05:10:56.311106 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:56.311084    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7688455878-p5cqw"
Nov 01 05:10:56.374294 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:56.374258    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e2e3aa4-158a-421e-bc20-4ad0a69b6ba6-maas-api-tls\") pod \"maas-api-7688455878-p5cqw\" (UID: \"5e2e3aa4-158a-421e-bc20-4ad0a69b6ba6\") " pod="odh-ai-gateway-infra/maas-api-7688455878-p5cqw"
Nov 01 05:10:56.374413 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:56.374324    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e2e3aa4-158a-421e-bc20-4ad0a69b6ba6-openshift-service-ca\") pod \"maas-api-7688455878-p5cqw\" (UID: \"5e2e3aa4-158a-421e-bc20-4ad0a69b6ba6\") " pod="odh-ai-gateway-infra/maas-api-7688455878-p5cqw"
Nov 01 05:10:56.374413 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:56.374395    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzqd6\" (UniqueName: \"kubernetes.io/projected/5e2e3aa4-158a-421e-bc20-4ad0a69b6ba6-kube-api-access-jzqd6\") pod \"maas-api-7688455878-p5cqw\" (UID: \"5e2e3aa4-158a-421e-bc20-4ad0a69b6ba6\") " pod="odh-ai-gateway-infra/maas-api-7688455878-p5cqw"
Nov 01 05:10:56.374519 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:56.374496    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e2e3aa4-158a-421e-bc20-4ad0a69b6ba6-metrics-certs\") pod \"maas-api-7688455878-p5cqw\" (UID: \"5e2e3aa4-158a-421e-bc20-4ad0a69b6ba6\") " pod="odh-ai-gateway-infra/maas-api-7688455878-p5cqw"
Nov 01 05:10:56.412440 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:56.412403    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 01 05:10:56.412580 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:56.412523    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9729w\""
Nov 01 05:10:56.412580 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:56.412537    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Nov 01 05:10:56.413440 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:56.413422    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7688455878-p5cqw"]
Nov 01 05:10:56.475382 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:56.475352    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e2e3aa4-158a-421e-bc20-4ad0a69b6ba6-openshift-service-ca\") pod \"maas-api-7688455878-p5cqw\" (UID: \"5e2e3aa4-158a-421e-bc20-4ad0a69b6ba6\") " pod="odh-ai-gateway-infra/maas-api-7688455878-p5cqw"
Nov 01 05:10:56.475382 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:56.475385    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzqd6\" (UniqueName: \"kubernetes.io/projected/5e2e3aa4-158a-421e-bc20-4ad0a69b6ba6-kube-api-access-jzqd6\") pod \"maas-api-7688455878-p5cqw\" (UID: \"5e2e3aa4-158a-421e-bc20-4ad0a69b6ba6\") " pod="odh-ai-gateway-infra/maas-api-7688455878-p5cqw"
Nov 01 05:10:56.475597 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:56.475419    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e2e3aa4-158a-421e-bc20-4ad0a69b6ba6-metrics-certs\") pod \"maas-api-7688455878-p5cqw\" (UID: \"5e2e3aa4-158a-421e-bc20-4ad0a69b6ba6\") " pod="odh-ai-gateway-infra/maas-api-7688455878-p5cqw"
Nov 01 05:10:56.475597 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:56.475492    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e2e3aa4-158a-421e-bc20-4ad0a69b6ba6-maas-api-tls\") pod \"maas-api-7688455878-p5cqw\" (UID: \"5e2e3aa4-158a-421e-bc20-4ad0a69b6ba6\") " pod="odh-ai-gateway-infra/maas-api-7688455878-p5cqw"
Nov 01 05:10:56.476179 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:56.476087    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e2e3aa4-158a-421e-bc20-4ad0a69b6ba6-openshift-service-ca\") pod \"maas-api-7688455878-p5cqw\" (UID: \"5e2e3aa4-158a-421e-bc20-4ad0a69b6ba6\") " pod="odh-ai-gateway-infra/maas-api-7688455878-p5cqw"
Nov 01 05:10:56.477868 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:56.477844    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e2e3aa4-158a-421e-bc20-4ad0a69b6ba6-metrics-certs\") pod \"maas-api-7688455878-p5cqw\" (UID: \"5e2e3aa4-158a-421e-bc20-4ad0a69b6ba6\") " pod="odh-ai-gateway-infra/maas-api-7688455878-p5cqw"
Nov 01 05:10:56.477965 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:56.477951    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e2e3aa4-158a-421e-bc20-4ad0a69b6ba6-maas-api-tls\") pod \"maas-api-7688455878-p5cqw\" (UID: \"5e2e3aa4-158a-421e-bc20-4ad0a69b6ba6\") " pod="odh-ai-gateway-infra/maas-api-7688455878-p5cqw"
Nov 01 05:10:56.535967 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:56.535942    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzqd6\" (UniqueName: \"kubernetes.io/projected/5e2e3aa4-158a-421e-bc20-4ad0a69b6ba6-kube-api-access-jzqd6\") pod \"maas-api-7688455878-p5cqw\" (UID: \"5e2e3aa4-158a-421e-bc20-4ad0a69b6ba6\") " pod="odh-ai-gateway-infra/maas-api-7688455878-p5cqw"
Nov 01 05:10:56.623754 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:56.623679    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7688455878-p5cqw"
Nov 01 05:10:56.995419 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:56.995389    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7688455878-p5cqw"]
Nov 01 05:10:56.998780 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:10:56.998752    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e2e3aa4_158a_421e_bc20_4ad0a69b6ba6.slice/crio-795fa3cda9c5a982b6968232419fa9edec51299469358b4e3951a2e70e681f33 WatchSource:0}: Error finding container 795fa3cda9c5a982b6968232419fa9edec51299469358b4e3951a2e70e681f33: Status 404 returned error can't find the container with id 795fa3cda9c5a982b6968232419fa9edec51299469358b4e3951a2e70e681f33
Nov 01 05:10:57.751111 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:10:57.751072    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7688455878-p5cqw" event={"ID":"5e2e3aa4-158a-421e-bc20-4ad0a69b6ba6","Type":"ContainerStarted","Data":"795fa3cda9c5a982b6968232419fa9edec51299469358b4e3951a2e70e681f33"}
Nov 01 05:11:00.069309 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:11:00.069278    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:11:00.721974 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:11:00.721936    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-pjms7"
Nov 01 05:11:01.791194 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:11:01.790554    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:11:04.784137 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:11:04.784100    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7688455878-p5cqw" event={"ID":"5e2e3aa4-158a-421e-bc20-4ad0a69b6ba6","Type":"ContainerStarted","Data":"de3869d9d8c32b9cf8cc254aa856523079932a5b619514f9dc72aaeed1d877ea"}
Nov 01 05:11:04.784612 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:11:04.784235    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7688455878-p5cqw"
Nov 01 05:11:04.902526 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:11:04.902441    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7688455878-p5cqw" podStartSLOduration=1.807565258 podStartE2EDuration="8.902421085s" totalImagesPullingTime="7.094855827s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:10:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:11:04.901064606 +0000 UTC m=+442.971861327" watchObservedRunningTime="2026-11-01 05:11:04.902421085 +0000 UTC m=+442.973217806"
Nov 01 05:11:08.043718 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:11:08.043681    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:11:08.219985 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:11:08.219944    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:11:10.794002 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:11:10.793975    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7688455878-p5cqw"
Nov 01 05:11:11.380699 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:11:11.380655    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:11:11.579836 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:11:11.579801    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:11:12.058436 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:11:12.058399    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5584c7d87f-wjpw2"]
Nov 01 05:11:12.064173 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:11:12.064149    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5584c7d87f-wjpw2"
Nov 01 05:11:12.120239 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:11:12.120213    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5584c7d87f-wjpw2"]
Nov 01 05:11:12.168597 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:11:12.168570    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-gq664\""
Nov 01 05:11:12.205073 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:11:12.205044    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdmhf\" (UniqueName: \"kubernetes.io/projected/00e792d0-f3c5-423c-9b14-43f13dc55116-kube-api-access-fdmhf\") pod \"authorino-5584c7d87f-wjpw2\" (UID: \"00e792d0-f3c5-423c-9b14-43f13dc55116\") " pod="kuadrant-system/authorino-5584c7d87f-wjpw2"
Nov 01 05:11:12.305668 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:11:12.305643    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdmhf\" (UniqueName: \"kubernetes.io/projected/00e792d0-f3c5-423c-9b14-43f13dc55116-kube-api-access-fdmhf\") pod \"authorino-5584c7d87f-wjpw2\" (UID: \"00e792d0-f3c5-423c-9b14-43f13dc55116\") " pod="kuadrant-system/authorino-5584c7d87f-wjpw2"
Nov 01 05:11:12.368120 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:11:12.368052    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdmhf\" (UniqueName: \"kubernetes.io/projected/00e792d0-f3c5-423c-9b14-43f13dc55116-kube-api-access-fdmhf\") pod \"authorino-5584c7d87f-wjpw2\" (UID: \"00e792d0-f3c5-423c-9b14-43f13dc55116\") " pod="kuadrant-system/authorino-5584c7d87f-wjpw2"
Nov 01 05:11:12.384730 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:11:12.384704    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5584c7d87f-wjpw2"
Nov 01 05:11:12.500333 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:11:12.500301    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5584c7d87f-wjpw2"]
Nov 01 05:11:12.758660 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:11:12.758631    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5584c7d87f-wjpw2"]
Nov 01 05:11:12.762646 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:11:12.762615    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod00e792d0_f3c5_423c_9b14_43f13dc55116.slice/crio-7aa23cb801f971d3df17b905b2b3ff385fa17acea5676e316ca23ce908d98141 WatchSource:0}: Error finding container 7aa23cb801f971d3df17b905b2b3ff385fa17acea5676e316ca23ce908d98141: Status 404 returned error can't find the container with id 7aa23cb801f971d3df17b905b2b3ff385fa17acea5676e316ca23ce908d98141
Nov 01 05:11:12.817184 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:11:12.817157    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5584c7d87f-wjpw2" event={"ID":"00e792d0-f3c5-423c-9b14-43f13dc55116","Type":"ContainerStarted","Data":"7aa23cb801f971d3df17b905b2b3ff385fa17acea5676e316ca23ce908d98141"}
Nov 01 05:11:14.828354 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:11:14.828316    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5584c7d87f-wjpw2" event={"ID":"00e792d0-f3c5-423c-9b14-43f13dc55116","Type":"ContainerStarted","Data":"36c6af93e1ab58ec52e53c746ce3c07c8ff90a76f335df5625b619705c53fdd2"}
Nov 01 05:11:14.828849 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:11:14.828354    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-5584c7d87f-wjpw2" podUID="00e792d0-f3c5-423c-9b14-43f13dc55116" containerName="authorino" containerID="cri-o://36c6af93e1ab58ec52e53c746ce3c07c8ff90a76f335df5625b619705c53fdd2" gracePeriod=30
Nov 01 05:11:14.963910 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:11:14.963851    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-5584c7d87f-wjpw2" podStartSLOduration=1.699908519 podStartE2EDuration="2.963835585s" totalImagesPullingTime="1.263927066s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:11:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:11:14.963338818 +0000 UTC m=+453.034135538" watchObservedRunningTime="2026-11-01 05:11:14.963835585 +0000 UTC m=+453.034632308"
Nov 01 05:11:15.132950 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:11:15.132922    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5584c7d87f-wjpw2"
Nov 01 05:11:15.229802 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:11:15.229769    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00e792d0-f3c5-423c-9b14-43f13dc55116-kube-api-access-fdmhf\" (UniqueName: \"kubernetes.io/projected/00e792d0-f3c5-423c-9b14-43f13dc55116-kube-api-access-fdmhf\") pod \"00e792d0-f3c5-423c-9b14-43f13dc55116\" (UID: \"00e792d0-f3c5-423c-9b14-43f13dc55116\") "
Nov 01 05:11:15.231909 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:11:15.231876    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00e792d0-f3c5-423c-9b14-43f13dc55116-kube-api-access-fdmhf" pod "00e792d0-f3c5-423c-9b14-43f13dc55116" (UID: "00e792d0-f3c5-423c-9b14-43f13dc55116"). InnerVolumeSpecName "kube-api-access-fdmhf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:11:15.330779 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:11:15.330745    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdmhf\" (UniqueName: \"kubernetes.io/projected/00e792d0-f3c5-423c-9b14-43f13dc55116-kube-api-access-fdmhf\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:11:15.833502 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:11:15.833444    5689 generic.go:356] "Generic (PLEG): container finished" podID="00e792d0-f3c5-423c-9b14-43f13dc55116" containerID="36c6af93e1ab58ec52e53c746ce3c07c8ff90a76f335df5625b619705c53fdd2" exitCode=0
Nov 01 05:11:15.833938 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:11:15.833507    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5584c7d87f-wjpw2"
Nov 01 05:11:15.833938 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:11:15.833521    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5584c7d87f-wjpw2" event={"ID":"00e792d0-f3c5-423c-9b14-43f13dc55116","Type":"ContainerDied","Data":"36c6af93e1ab58ec52e53c746ce3c07c8ff90a76f335df5625b619705c53fdd2"}
Nov 01 05:11:15.833938 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:11:15.833552    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5584c7d87f-wjpw2" event={"ID":"00e792d0-f3c5-423c-9b14-43f13dc55116","Type":"ContainerDied","Data":"7aa23cb801f971d3df17b905b2b3ff385fa17acea5676e316ca23ce908d98141"}
Nov 01 05:11:15.833938 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:11:15.833569    5689 scope.go:122] "RemoveContainer" containerID="36c6af93e1ab58ec52e53c746ce3c07c8ff90a76f335df5625b619705c53fdd2"
Nov 01 05:11:15.841906 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:11:15.841890    5689 scope.go:122] "RemoveContainer" containerID="36c6af93e1ab58ec52e53c746ce3c07c8ff90a76f335df5625b619705c53fdd2"
Nov 01 05:11:15.842137 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:11:15.842118    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36c6af93e1ab58ec52e53c746ce3c07c8ff90a76f335df5625b619705c53fdd2\": container with ID starting with 36c6af93e1ab58ec52e53c746ce3c07c8ff90a76f335df5625b619705c53fdd2 not found: ID does not exist" containerID="36c6af93e1ab58ec52e53c746ce3c07c8ff90a76f335df5625b619705c53fdd2"
Nov 01 05:11:15.842201 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:11:15.842151    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36c6af93e1ab58ec52e53c746ce3c07c8ff90a76f335df5625b619705c53fdd2"} err="failed to get container status \"36c6af93e1ab58ec52e53c746ce3c07c8ff90a76f335df5625b619705c53fdd2\": rpc error: code = NotFound desc = could not find container \"36c6af93e1ab58ec52e53c746ce3c07c8ff90a76f335df5625b619705c53fdd2\": container with ID starting with 36c6af93e1ab58ec52e53c746ce3c07c8ff90a76f335df5625b619705c53fdd2 not found: ID does not exist"
Nov 01 05:11:15.999058 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:11:15.999027    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5584c7d87f-wjpw2"]
Nov 01 05:11:16.003119 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:11:16.003095    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5584c7d87f-wjpw2"]
Nov 01 05:11:16.066994 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:11:16.066968    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00e792d0-f3c5-423c-9b14-43f13dc55116" path="/var/lib/kubelet/pods/00e792d0-f3c5-423c-9b14-43f13dc55116/volumes"
Nov 01 05:11:18.716503 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:11:18.716471    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-hstph"
Nov 01 05:11:21.087861 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:11:21.087789    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7688455878-p5cqw"]
Nov 01 05:11:21.088245 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:11:21.087973    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-7688455878-p5cqw" podUID="5e2e3aa4-158a-421e-bc20-4ad0a69b6ba6" containerName="maas-api" containerID="cri-o://de3869d9d8c32b9cf8cc254aa856523079932a5b619514f9dc72aaeed1d877ea" gracePeriod=30
Nov 01 05:11:21.325847 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:11:21.325818    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7688455878-p5cqw"
Nov 01 05:11:21.477955 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:11:21.477921    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e2e3aa4-158a-421e-bc20-4ad0a69b6ba6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e2e3aa4-158a-421e-bc20-4ad0a69b6ba6-openshift-service-ca\") pod \"5e2e3aa4-158a-421e-bc20-4ad0a69b6ba6\" (UID: \"5e2e3aa4-158a-421e-bc20-4ad0a69b6ba6\") "
Nov 01 05:11:21.478126 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:11:21.477967    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e2e3aa4-158a-421e-bc20-4ad0a69b6ba6-kube-api-access-jzqd6\" (UniqueName: \"kubernetes.io/projected/5e2e3aa4-158a-421e-bc20-4ad0a69b6ba6-kube-api-access-jzqd6\") pod \"5e2e3aa4-158a-421e-bc20-4ad0a69b6ba6\" (UID: \"5e2e3aa4-158a-421e-bc20-4ad0a69b6ba6\") "
Nov 01 05:11:21.478126 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:11:21.478049    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e2e3aa4-158a-421e-bc20-4ad0a69b6ba6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e2e3aa4-158a-421e-bc20-4ad0a69b6ba6-maas-api-tls\") pod \"5e2e3aa4-158a-421e-bc20-4ad0a69b6ba6\" (UID: \"5e2e3aa4-158a-421e-bc20-4ad0a69b6ba6\") "
Nov 01 05:11:21.478126 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:11:21.478066    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e2e3aa4-158a-421e-bc20-4ad0a69b6ba6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e2e3aa4-158a-421e-bc20-4ad0a69b6ba6-metrics-certs\") pod \"5e2e3aa4-158a-421e-bc20-4ad0a69b6ba6\" (UID: \"5e2e3aa4-158a-421e-bc20-4ad0a69b6ba6\") "
Nov 01 05:11:21.478306 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:11:21.478271    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e2e3aa4-158a-421e-bc20-4ad0a69b6ba6-openshift-service-ca" pod "5e2e3aa4-158a-421e-bc20-4ad0a69b6ba6" (UID: "5e2e3aa4-158a-421e-bc20-4ad0a69b6ba6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:11:21.480333 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:11:21.480301    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e2e3aa4-158a-421e-bc20-4ad0a69b6ba6-maas-api-tls" pod "5e2e3aa4-158a-421e-bc20-4ad0a69b6ba6" (UID: "5e2e3aa4-158a-421e-bc20-4ad0a69b6ba6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 05:11:21.480496 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:11:21.480340    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e2e3aa4-158a-421e-bc20-4ad0a69b6ba6-metrics-certs" pod "5e2e3aa4-158a-421e-bc20-4ad0a69b6ba6" (UID: "5e2e3aa4-158a-421e-bc20-4ad0a69b6ba6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 05:11:21.480496 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:11:21.480399    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e2e3aa4-158a-421e-bc20-4ad0a69b6ba6-kube-api-access-jzqd6" pod "5e2e3aa4-158a-421e-bc20-4ad0a69b6ba6" (UID: "5e2e3aa4-158a-421e-bc20-4ad0a69b6ba6"). InnerVolumeSpecName "kube-api-access-jzqd6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:11:21.578933 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:11:21.578903    5689 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e2e3aa4-158a-421e-bc20-4ad0a69b6ba6-maas-api-tls\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:11:21.578933 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:11:21.578926    5689 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e2e3aa4-158a-421e-bc20-4ad0a69b6ba6-metrics-certs\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:11:21.578933 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:11:21.578937    5689 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e2e3aa4-158a-421e-bc20-4ad0a69b6ba6-openshift-service-ca\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:11:21.579126 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:11:21.578945    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzqd6\" (UniqueName: \"kubernetes.io/projected/5e2e3aa4-158a-421e-bc20-4ad0a69b6ba6-kube-api-access-jzqd6\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:11:21.861160 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:11:21.861075    5689 generic.go:356] "Generic (PLEG): container finished" podID="5e2e3aa4-158a-421e-bc20-4ad0a69b6ba6" containerID="de3869d9d8c32b9cf8cc254aa856523079932a5b619514f9dc72aaeed1d877ea" exitCode=0
Nov 01 05:11:21.861160 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:11:21.861128    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7688455878-p5cqw"
Nov 01 05:11:21.861362 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:11:21.861154    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7688455878-p5cqw" event={"ID":"5e2e3aa4-158a-421e-bc20-4ad0a69b6ba6","Type":"ContainerDied","Data":"de3869d9d8c32b9cf8cc254aa856523079932a5b619514f9dc72aaeed1d877ea"}
Nov 01 05:11:21.861362 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:11:21.861206    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7688455878-p5cqw" event={"ID":"5e2e3aa4-158a-421e-bc20-4ad0a69b6ba6","Type":"ContainerDied","Data":"795fa3cda9c5a982b6968232419fa9edec51299469358b4e3951a2e70e681f33"}
Nov 01 05:11:21.861362 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:11:21.861227    5689 scope.go:122] "RemoveContainer" containerID="de3869d9d8c32b9cf8cc254aa856523079932a5b619514f9dc72aaeed1d877ea"
Nov 01 05:11:21.870796 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:11:21.870782    5689 scope.go:122] "RemoveContainer" containerID="de3869d9d8c32b9cf8cc254aa856523079932a5b619514f9dc72aaeed1d877ea"
Nov 01 05:11:21.871060 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:11:21.871037    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de3869d9d8c32b9cf8cc254aa856523079932a5b619514f9dc72aaeed1d877ea\": container with ID starting with de3869d9d8c32b9cf8cc254aa856523079932a5b619514f9dc72aaeed1d877ea not found: ID does not exist" containerID="de3869d9d8c32b9cf8cc254aa856523079932a5b619514f9dc72aaeed1d877ea"
Nov 01 05:11:21.871121 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:11:21.871071    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de3869d9d8c32b9cf8cc254aa856523079932a5b619514f9dc72aaeed1d877ea"} err="failed to get container status \"de3869d9d8c32b9cf8cc254aa856523079932a5b619514f9dc72aaeed1d877ea\": rpc error: code = NotFound desc = could not find container \"de3869d9d8c32b9cf8cc254aa856523079932a5b619514f9dc72aaeed1d877ea\": container with ID starting with de3869d9d8c32b9cf8cc254aa856523079932a5b619514f9dc72aaeed1d877ea not found: ID does not exist"
Nov 01 05:11:22.023865 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:11:22.023838    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7688455878-p5cqw"]
Nov 01 05:11:22.073243 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:11:22.073219    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-7688455878-p5cqw"]
Nov 01 05:11:24.066384 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:11:24.066347    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e2e3aa4-158a-421e-bc20-4ad0a69b6ba6" path="/var/lib/kubelet/pods/5e2e3aa4-158a-421e-bc20-4ad0a69b6ba6/volumes"
Nov 01 05:11:30.067331 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:11:30.067299    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:11:30.875429 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:11:30.875391    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:11:56.192933 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:11:56.192887    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:11:56.364564 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:11:56.364534    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:12:00.066983 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:12:00.066946    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:12:12.293684 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:12:12.293651    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:12:12.468021 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:12:12.467981    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:12:17.568788 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:12:17.568721    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:12:30.068849 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:12:30.068813    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:12:30.447008 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:12:30.446977    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:12:30.640623 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:12:30.640576    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:12:38.009049 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:12:38.009021    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:12:38.192210 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:12:38.192182    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:12:56.542846 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:12:56.542764    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:13:00.066383 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:13:00.066350    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:13:00.198595 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:00.198563    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:13:00.786754 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:00.786719    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:13:02.046775 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:02.046743    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:13:02.234921 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:02.234889    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:13:10.616630 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:10.616581    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-6h4ll"]
Nov 01 05:13:10.617409 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:10.617387    5689 cpu_manager.go:414] "containerMap: removing container" podUID="5e2e3aa4-158a-421e-bc20-4ad0a69b6ba6" containerName="maas-api"
Nov 01 05:13:10.617560 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:10.617411    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e2e3aa4-158a-421e-bc20-4ad0a69b6ba6" containerName="maas-api"
Nov 01 05:13:10.617560 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:10.617500    5689 cpu_manager.go:414] "containerMap: removing container" podUID="00e792d0-f3c5-423c-9b14-43f13dc55116" containerName="authorino"
Nov 01 05:13:10.617560 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:10.617510    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00e792d0-f3c5-423c-9b14-43f13dc55116" containerName="authorino"
Nov 01 05:13:10.617730 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:10.617716    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="00e792d0-f3c5-423c-9b14-43f13dc55116" containerName="authorino"
Nov 01 05:13:10.617782 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:10.617741    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e2e3aa4-158a-421e-bc20-4ad0a69b6ba6" containerName="maas-api"
Nov 01 05:13:10.621079 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:10.621057    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-6h4ll"
Nov 01 05:13:10.701336 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:10.701304    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Nov 01 05:13:10.723213 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:10.723180    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c142295-167c-45dc-8287-cce5bb7826c8-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-6h4ll\" (UID: \"9c142295-167c-45dc-8287-cce5bb7826c8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-6h4ll"
Nov 01 05:13:10.723343 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:10.723229    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2fbx\" (UniqueName: \"kubernetes.io/projected/9c142295-167c-45dc-8287-cce5bb7826c8-kube-api-access-k2fbx\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-6h4ll\" (UID: \"9c142295-167c-45dc-8287-cce5bb7826c8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-6h4ll"
Nov 01 05:13:10.723408 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:10.723344    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c142295-167c-45dc-8287-cce5bb7826c8-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-6h4ll\" (UID: \"9c142295-167c-45dc-8287-cce5bb7826c8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-6h4ll"
Nov 01 05:13:10.723494 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:10.723412    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c142295-167c-45dc-8287-cce5bb7826c8-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-6h4ll\" (UID: \"9c142295-167c-45dc-8287-cce5bb7826c8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-6h4ll"
Nov 01 05:13:10.723494 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:10.723441    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c142295-167c-45dc-8287-cce5bb7826c8-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-6h4ll\" (UID: \"9c142295-167c-45dc-8287-cce5bb7826c8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-6h4ll"
Nov 01 05:13:10.723494 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:10.723485    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-6w97n\""
Nov 01 05:13:10.723603 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:10.723542    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c142295-167c-45dc-8287-cce5bb7826c8-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-6h4ll\" (UID: \"9c142295-167c-45dc-8287-cce5bb7826c8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-6h4ll"
Nov 01 05:13:10.723724 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:10.723709    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Nov 01 05:13:10.723771 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:10.723711    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Nov 01 05:13:10.724702 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:10.724685    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-6h4ll"]
Nov 01 05:13:10.744638 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:10.744611    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:13:10.824761 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:10.824733    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c142295-167c-45dc-8287-cce5bb7826c8-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-6h4ll\" (UID: \"9c142295-167c-45dc-8287-cce5bb7826c8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-6h4ll"
Nov 01 05:13:10.824861 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:10.824764    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c142295-167c-45dc-8287-cce5bb7826c8-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-6h4ll\" (UID: \"9c142295-167c-45dc-8287-cce5bb7826c8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-6h4ll"
Nov 01 05:13:10.824861 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:10.824788    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c142295-167c-45dc-8287-cce5bb7826c8-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-6h4ll\" (UID: \"9c142295-167c-45dc-8287-cce5bb7826c8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-6h4ll"
Nov 01 05:13:10.824861 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:10.824833    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c142295-167c-45dc-8287-cce5bb7826c8-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-6h4ll\" (UID: \"9c142295-167c-45dc-8287-cce5bb7826c8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-6h4ll"
Nov 01 05:13:10.824861 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:10.824854    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2fbx\" (UniqueName: \"kubernetes.io/projected/9c142295-167c-45dc-8287-cce5bb7826c8-kube-api-access-k2fbx\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-6h4ll\" (UID: \"9c142295-167c-45dc-8287-cce5bb7826c8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-6h4ll"
Nov 01 05:13:10.825046 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:10.824873    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c142295-167c-45dc-8287-cce5bb7826c8-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-6h4ll\" (UID: \"9c142295-167c-45dc-8287-cce5bb7826c8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-6h4ll"
Nov 01 05:13:10.825236 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:10.825211    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c142295-167c-45dc-8287-cce5bb7826c8-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-6h4ll\" (UID: \"9c142295-167c-45dc-8287-cce5bb7826c8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-6h4ll"
Nov 01 05:13:10.825312 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:10.825256    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c142295-167c-45dc-8287-cce5bb7826c8-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-6h4ll\" (UID: \"9c142295-167c-45dc-8287-cce5bb7826c8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-6h4ll"
Nov 01 05:13:10.825368 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:10.825328    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c142295-167c-45dc-8287-cce5bb7826c8-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-6h4ll\" (UID: \"9c142295-167c-45dc-8287-cce5bb7826c8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-6h4ll"
Nov 01 05:13:10.827161 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:10.827143    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c142295-167c-45dc-8287-cce5bb7826c8-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-6h4ll\" (UID: \"9c142295-167c-45dc-8287-cce5bb7826c8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-6h4ll"
Nov 01 05:13:10.827409 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:10.827391    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c142295-167c-45dc-8287-cce5bb7826c8-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-6h4ll\" (UID: \"9c142295-167c-45dc-8287-cce5bb7826c8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-6h4ll"
Nov 01 05:13:10.885698 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:10.885642    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2fbx\" (UniqueName: \"kubernetes.io/projected/9c142295-167c-45dc-8287-cce5bb7826c8-kube-api-access-k2fbx\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-6h4ll\" (UID: \"9c142295-167c-45dc-8287-cce5bb7826c8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-6h4ll"
Nov 01 05:13:10.934577 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:10.934552    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-6h4ll"
Nov 01 05:13:11.308352 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:11.308324    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-6h4ll"]
Nov 01 05:13:11.311906 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:13:11.311878    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c142295_167c_45dc_8287_cce5bb7826c8.slice/crio-4a8daf71c1a50b2535f53a0a731d70220e65108e592311df6bdcccd9bfdc13bd WatchSource:0}: Error finding container 4a8daf71c1a50b2535f53a0a731d70220e65108e592311df6bdcccd9bfdc13bd: Status 404 returned error can't find the container with id 4a8daf71c1a50b2535f53a0a731d70220e65108e592311df6bdcccd9bfdc13bd
Nov 01 05:13:12.295393 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:12.295350    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-6h4ll" event={"ID":"9c142295-167c-45dc-8287-cce5bb7826c8","Type":"ContainerStarted","Data":"4a8daf71c1a50b2535f53a0a731d70220e65108e592311df6bdcccd9bfdc13bd"}
Nov 01 05:13:14.306096 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:14.306062    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-6h4ll" event={"ID":"9c142295-167c-45dc-8287-cce5bb7826c8","Type":"ContainerStarted","Data":"47dfa4cf19ce206bfbfdc81af3138f6ddeb7bdf967f1644b30bded85cfae4857"}
Nov 01 05:13:14.423573 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:14.423519    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-6h4ll" podStartSLOduration=2.206277464 podStartE2EDuration="4.423506794s" totalImagesPullingTime="2.21722933s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:13:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:13:14.423135783 +0000 UTC m=+572.493932498" watchObservedRunningTime="2026-11-01 05:13:14.423506794 +0000 UTC m=+572.494303508"
Nov 01 05:13:14.934766 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:14.934726    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-6h4ll"
Nov 01 05:13:15.561070 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:15.561028    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:13:20.667282 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:20.667246    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:13:20.935316 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:20.935239    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-6h4ll"
Nov 01 05:13:20.947314 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:20.947287    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-6h4ll"
Nov 01 05:13:21.344146 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:21.344075    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-6h4ll"
Nov 01 05:13:24.083797 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:24.083761    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-d797t"]
Nov 01 05:13:24.086988 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:24.086972    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-d797t"
Nov 01 05:13:24.146586 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:24.146559    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-d797t"]
Nov 01 05:13:24.188049 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:24.188025    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Nov 01 05:13:24.208765 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:24.208731    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:13:24.239201 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:24.239174    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7109ff45-d2c8-42ce-9196-931fc105574c-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-d797t\" (UID: \"7109ff45-d2c8-42ce-9196-931fc105574c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-d797t"
Nov 01 05:13:24.239353 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:24.239219    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7109ff45-d2c8-42ce-9196-931fc105574c-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-d797t\" (UID: \"7109ff45-d2c8-42ce-9196-931fc105574c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-d797t"
Nov 01 05:13:24.239353 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:24.239237    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7109ff45-d2c8-42ce-9196-931fc105574c-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-d797t\" (UID: \"7109ff45-d2c8-42ce-9196-931fc105574c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-d797t"
Nov 01 05:13:24.239353 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:24.239339    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7109ff45-d2c8-42ce-9196-931fc105574c-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-d797t\" (UID: \"7109ff45-d2c8-42ce-9196-931fc105574c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-d797t"
Nov 01 05:13:24.239537 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:24.239395    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7109ff45-d2c8-42ce-9196-931fc105574c-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-d797t\" (UID: \"7109ff45-d2c8-42ce-9196-931fc105574c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-d797t"
Nov 01 05:13:24.239537 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:24.239442    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9q7q\" (UniqueName: \"kubernetes.io/projected/7109ff45-d2c8-42ce-9196-931fc105574c-kube-api-access-b9q7q\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-d797t\" (UID: \"7109ff45-d2c8-42ce-9196-931fc105574c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-d797t"
Nov 01 05:13:24.340295 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:24.340218    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7109ff45-d2c8-42ce-9196-931fc105574c-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-d797t\" (UID: \"7109ff45-d2c8-42ce-9196-931fc105574c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-d797t"
Nov 01 05:13:24.340295 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:24.340275    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7109ff45-d2c8-42ce-9196-931fc105574c-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-d797t\" (UID: \"7109ff45-d2c8-42ce-9196-931fc105574c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-d797t"
Nov 01 05:13:24.340524 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:24.340300    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9q7q\" (UniqueName: \"kubernetes.io/projected/7109ff45-d2c8-42ce-9196-931fc105574c-kube-api-access-b9q7q\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-d797t\" (UID: \"7109ff45-d2c8-42ce-9196-931fc105574c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-d797t"
Nov 01 05:13:24.340524 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:24.340350    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7109ff45-d2c8-42ce-9196-931fc105574c-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-d797t\" (UID: \"7109ff45-d2c8-42ce-9196-931fc105574c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-d797t"
Nov 01 05:13:24.340524 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:24.340401    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7109ff45-d2c8-42ce-9196-931fc105574c-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-d797t\" (UID: \"7109ff45-d2c8-42ce-9196-931fc105574c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-d797t"
Nov 01 05:13:24.340524 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:24.340426    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7109ff45-d2c8-42ce-9196-931fc105574c-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-d797t\" (UID: \"7109ff45-d2c8-42ce-9196-931fc105574c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-d797t"
Nov 01 05:13:24.340723 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:24.340637    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7109ff45-d2c8-42ce-9196-931fc105574c-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-d797t\" (UID: \"7109ff45-d2c8-42ce-9196-931fc105574c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-d797t"
Nov 01 05:13:24.340774 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:24.340743    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7109ff45-d2c8-42ce-9196-931fc105574c-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-d797t\" (UID: \"7109ff45-d2c8-42ce-9196-931fc105574c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-d797t"
Nov 01 05:13:24.340825 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:24.340751    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7109ff45-d2c8-42ce-9196-931fc105574c-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-d797t\" (UID: \"7109ff45-d2c8-42ce-9196-931fc105574c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-d797t"
Nov 01 05:13:24.342658 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:24.342638    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7109ff45-d2c8-42ce-9196-931fc105574c-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-d797t\" (UID: \"7109ff45-d2c8-42ce-9196-931fc105574c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-d797t"
Nov 01 05:13:24.342907 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:24.342887    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7109ff45-d2c8-42ce-9196-931fc105574c-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-d797t\" (UID: \"7109ff45-d2c8-42ce-9196-931fc105574c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-d797t"
Nov 01 05:13:24.401048 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:24.401014    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9q7q\" (UniqueName: \"kubernetes.io/projected/7109ff45-d2c8-42ce-9196-931fc105574c-kube-api-access-b9q7q\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-d797t\" (UID: \"7109ff45-d2c8-42ce-9196-931fc105574c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-d797t"
Nov 01 05:13:24.698869 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:24.698831    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-d797t"
Nov 01 05:13:25.075261 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:25.075187    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-d797t"]
Nov 01 05:13:25.078392 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:13:25.078361    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7109ff45_d2c8_42ce_9196_931fc105574c.slice/crio-37bca6268d81d70ebee8f8b54ca0e2180e01aecd584a1d57c7e797bc20886db6 WatchSource:0}: Error finding container 37bca6268d81d70ebee8f8b54ca0e2180e01aecd584a1d57c7e797bc20886db6: Status 404 returned error can't find the container with id 37bca6268d81d70ebee8f8b54ca0e2180e01aecd584a1d57c7e797bc20886db6
Nov 01 05:13:25.352810 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:25.352740    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-d797t" event={"ID":"7109ff45-d2c8-42ce-9196-931fc105574c","Type":"ContainerStarted","Data":"37bca6268d81d70ebee8f8b54ca0e2180e01aecd584a1d57c7e797bc20886db6"}
Nov 01 05:13:25.494338 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:25.494311    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jxzw"]
Nov 01 05:13:25.498833 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:25.498813    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jxzw"
Nov 01 05:13:25.551366 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:25.551334    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-8hgdn\""
Nov 01 05:13:25.601571 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:25.601543    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jxzw"]
Nov 01 05:13:25.652222 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:25.652197    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54667545-c5d3-4893-9ade-e16c82cb5b93-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-4jxzw\" (UID: \"54667545-c5d3-4893-9ade-e16c82cb5b93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jxzw"
Nov 01 05:13:25.652367 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:25.652229    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdkdm\" (UniqueName: \"kubernetes.io/projected/54667545-c5d3-4893-9ade-e16c82cb5b93-kube-api-access-fdkdm\") pod \"maas-default-gateway-openshift-default-58d898bd55-4jxzw\" (UID: \"54667545-c5d3-4893-9ade-e16c82cb5b93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jxzw"
Nov 01 05:13:25.652367 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:25.652249    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54667545-c5d3-4893-9ade-e16c82cb5b93-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-4jxzw\" (UID: \"54667545-c5d3-4893-9ade-e16c82cb5b93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jxzw"
Nov 01 05:13:25.652367 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:25.652350    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54667545-c5d3-4893-9ade-e16c82cb5b93-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-4jxzw\" (UID: \"54667545-c5d3-4893-9ade-e16c82cb5b93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jxzw"
Nov 01 05:13:25.652538 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:25.652381    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54667545-c5d3-4893-9ade-e16c82cb5b93-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-4jxzw\" (UID: \"54667545-c5d3-4893-9ade-e16c82cb5b93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jxzw"
Nov 01 05:13:25.652538 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:25.652408    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54667545-c5d3-4893-9ade-e16c82cb5b93-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-4jxzw\" (UID: \"54667545-c5d3-4893-9ade-e16c82cb5b93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jxzw"
Nov 01 05:13:25.652538 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:25.652429    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54667545-c5d3-4893-9ade-e16c82cb5b93-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-4jxzw\" (UID: \"54667545-c5d3-4893-9ade-e16c82cb5b93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jxzw"
Nov 01 05:13:25.652538 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:25.652514    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54667545-c5d3-4893-9ade-e16c82cb5b93-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-4jxzw\" (UID: \"54667545-c5d3-4893-9ade-e16c82cb5b93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jxzw"
Nov 01 05:13:25.652717 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:25.652684    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54667545-c5d3-4893-9ade-e16c82cb5b93-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-4jxzw\" (UID: \"54667545-c5d3-4893-9ade-e16c82cb5b93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jxzw"
Nov 01 05:13:25.753839 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:25.753795    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54667545-c5d3-4893-9ade-e16c82cb5b93-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-4jxzw\" (UID: \"54667545-c5d3-4893-9ade-e16c82cb5b93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jxzw"
Nov 01 05:13:25.753839 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:25.753836    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54667545-c5d3-4893-9ade-e16c82cb5b93-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-4jxzw\" (UID: \"54667545-c5d3-4893-9ade-e16c82cb5b93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jxzw"
Nov 01 05:13:25.754041 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:25.753855    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54667545-c5d3-4893-9ade-e16c82cb5b93-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-4jxzw\" (UID: \"54667545-c5d3-4893-9ade-e16c82cb5b93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jxzw"
Nov 01 05:13:25.754041 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:25.753877    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54667545-c5d3-4893-9ade-e16c82cb5b93-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-4jxzw\" (UID: \"54667545-c5d3-4893-9ade-e16c82cb5b93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jxzw"
Nov 01 05:13:25.754041 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:25.753896    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54667545-c5d3-4893-9ade-e16c82cb5b93-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-4jxzw\" (UID: \"54667545-c5d3-4893-9ade-e16c82cb5b93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jxzw"
Nov 01 05:13:25.754041 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:25.753916    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54667545-c5d3-4893-9ade-e16c82cb5b93-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-4jxzw\" (UID: \"54667545-c5d3-4893-9ade-e16c82cb5b93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jxzw"
Nov 01 05:13:25.754041 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:25.753953    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54667545-c5d3-4893-9ade-e16c82cb5b93-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-4jxzw\" (UID: \"54667545-c5d3-4893-9ade-e16c82cb5b93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jxzw"
Nov 01 05:13:25.754298 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:25.754032    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54667545-c5d3-4893-9ade-e16c82cb5b93-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-4jxzw\" (UID: \"54667545-c5d3-4893-9ade-e16c82cb5b93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jxzw"
Nov 01 05:13:25.754298 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:25.754070    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdkdm\" (UniqueName: \"kubernetes.io/projected/54667545-c5d3-4893-9ade-e16c82cb5b93-kube-api-access-fdkdm\") pod \"maas-default-gateway-openshift-default-58d898bd55-4jxzw\" (UID: \"54667545-c5d3-4893-9ade-e16c82cb5b93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jxzw"
Nov 01 05:13:25.754384 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:25.754316    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54667545-c5d3-4893-9ade-e16c82cb5b93-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-4jxzw\" (UID: \"54667545-c5d3-4893-9ade-e16c82cb5b93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jxzw"
Nov 01 05:13:25.754499 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:25.754471    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54667545-c5d3-4893-9ade-e16c82cb5b93-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-4jxzw\" (UID: \"54667545-c5d3-4893-9ade-e16c82cb5b93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jxzw"
Nov 01 05:13:25.754650 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:25.754547    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54667545-c5d3-4893-9ade-e16c82cb5b93-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-4jxzw\" (UID: \"54667545-c5d3-4893-9ade-e16c82cb5b93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jxzw"
Nov 01 05:13:25.754650 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:25.754600    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54667545-c5d3-4893-9ade-e16c82cb5b93-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-4jxzw\" (UID: \"54667545-c5d3-4893-9ade-e16c82cb5b93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jxzw"
Nov 01 05:13:25.754650 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:25.754627    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54667545-c5d3-4893-9ade-e16c82cb5b93-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-4jxzw\" (UID: \"54667545-c5d3-4893-9ade-e16c82cb5b93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jxzw"
Nov 01 05:13:25.756271 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:25.756248    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54667545-c5d3-4893-9ade-e16c82cb5b93-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-4jxzw\" (UID: \"54667545-c5d3-4893-9ade-e16c82cb5b93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jxzw"
Nov 01 05:13:25.756883 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:25.756863    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54667545-c5d3-4893-9ade-e16c82cb5b93-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-4jxzw\" (UID: \"54667545-c5d3-4893-9ade-e16c82cb5b93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jxzw"
Nov 01 05:13:25.814566 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:25.814539    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54667545-c5d3-4893-9ade-e16c82cb5b93-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-4jxzw\" (UID: \"54667545-c5d3-4893-9ade-e16c82cb5b93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jxzw"
Nov 01 05:13:25.814685 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:25.814616    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdkdm\" (UniqueName: \"kubernetes.io/projected/54667545-c5d3-4893-9ade-e16c82cb5b93-kube-api-access-fdkdm\") pod \"maas-default-gateway-openshift-default-58d898bd55-4jxzw\" (UID: \"54667545-c5d3-4893-9ade-e16c82cb5b93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jxzw"
Nov 01 05:13:26.111823 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:26.111757    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jxzw"
Nov 01 05:13:26.358955 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:26.358921    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-d797t" event={"ID":"7109ff45-d2c8-42ce-9196-931fc105574c","Type":"ContainerStarted","Data":"821db840f06fe6968fdd32b70eec6df657fd47e489d232f43b24960ef1451d4a"}
Nov 01 05:13:26.477932 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:26.477883    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-d797t" podStartSLOduration=1.8795807789999999 podStartE2EDuration="2.47787003s" totalImagesPullingTime="598.289251ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:13:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:13:26.477408977 +0000 UTC m=+584.548205698" watchObservedRunningTime="2026-11-01 05:13:26.47787003 +0000 UTC m=+584.548666749"
Nov 01 05:13:26.539512 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:26.539485    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jxzw"]
Nov 01 05:13:26.543287 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:13:26.543258    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54667545_c5d3_4893_9ade_e16c82cb5b93.slice/crio-4c608f66ee39e3256288a4bbeae1817343a1ee77419961706169a68bcf15ece2 WatchSource:0}: Error finding container 4c608f66ee39e3256288a4bbeae1817343a1ee77419961706169a68bcf15ece2: Status 404 returned error can't find the container with id 4c608f66ee39e3256288a4bbeae1817343a1ee77419961706169a68bcf15ece2
Nov 01 05:13:26.699140 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:26.699112    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-d797t"
Nov 01 05:13:27.364638 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:27.364604    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jxzw" event={"ID":"54667545-c5d3-4893-9ade-e16c82cb5b93","Type":"ContainerStarted","Data":"4c608f66ee39e3256288a4bbeae1817343a1ee77419961706169a68bcf15ece2"}
Nov 01 05:13:27.532133 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:27.532101    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-bpkrz"]
Nov 01 05:13:27.538094 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:27.538076    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-bpkrz"
Nov 01 05:13:27.590239 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:27.590212    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Nov 01 05:13:27.640413 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:27.640338    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-bpkrz"]
Nov 01 05:13:27.657032 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:27.657002    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:13:27.671184 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:27.671154    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bffb1fa8-30a6-47ef-8766-1efb36d2d8c4-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-bpkrz\" (UID: \"bffb1fa8-30a6-47ef-8766-1efb36d2d8c4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-bpkrz"
Nov 01 05:13:27.671273 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:27.671187    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bffb1fa8-30a6-47ef-8766-1efb36d2d8c4-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-bpkrz\" (UID: \"bffb1fa8-30a6-47ef-8766-1efb36d2d8c4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-bpkrz"
Nov 01 05:13:27.671273 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:27.671217    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bffb1fa8-30a6-47ef-8766-1efb36d2d8c4-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-bpkrz\" (UID: \"bffb1fa8-30a6-47ef-8766-1efb36d2d8c4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-bpkrz"
Nov 01 05:13:27.671384 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:27.671367    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctpq2\" (UniqueName: \"kubernetes.io/projected/bffb1fa8-30a6-47ef-8766-1efb36d2d8c4-kube-api-access-ctpq2\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-bpkrz\" (UID: \"bffb1fa8-30a6-47ef-8766-1efb36d2d8c4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-bpkrz"
Nov 01 05:13:27.671420 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:27.671403    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bffb1fa8-30a6-47ef-8766-1efb36d2d8c4-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-bpkrz\" (UID: \"bffb1fa8-30a6-47ef-8766-1efb36d2d8c4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-bpkrz"
Nov 01 05:13:27.671476 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:27.671421    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bffb1fa8-30a6-47ef-8766-1efb36d2d8c4-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-bpkrz\" (UID: \"bffb1fa8-30a6-47ef-8766-1efb36d2d8c4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-bpkrz"
Nov 01 05:13:27.771944 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:27.771913    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bffb1fa8-30a6-47ef-8766-1efb36d2d8c4-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-bpkrz\" (UID: \"bffb1fa8-30a6-47ef-8766-1efb36d2d8c4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-bpkrz"
Nov 01 05:13:27.772067 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:27.771949    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bffb1fa8-30a6-47ef-8766-1efb36d2d8c4-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-bpkrz\" (UID: \"bffb1fa8-30a6-47ef-8766-1efb36d2d8c4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-bpkrz"
Nov 01 05:13:27.772067 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:27.771973    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bffb1fa8-30a6-47ef-8766-1efb36d2d8c4-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-bpkrz\" (UID: \"bffb1fa8-30a6-47ef-8766-1efb36d2d8c4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-bpkrz"
Nov 01 05:13:27.772067 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:27.772026    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctpq2\" (UniqueName: \"kubernetes.io/projected/bffb1fa8-30a6-47ef-8766-1efb36d2d8c4-kube-api-access-ctpq2\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-bpkrz\" (UID: \"bffb1fa8-30a6-47ef-8766-1efb36d2d8c4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-bpkrz"
Nov 01 05:13:27.772067 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:27.772052    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bffb1fa8-30a6-47ef-8766-1efb36d2d8c4-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-bpkrz\" (UID: \"bffb1fa8-30a6-47ef-8766-1efb36d2d8c4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-bpkrz"
Nov 01 05:13:27.772293 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:27.772070    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bffb1fa8-30a6-47ef-8766-1efb36d2d8c4-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-bpkrz\" (UID: \"bffb1fa8-30a6-47ef-8766-1efb36d2d8c4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-bpkrz"
Nov 01 05:13:27.772362 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:27.772331    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bffb1fa8-30a6-47ef-8766-1efb36d2d8c4-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-bpkrz\" (UID: \"bffb1fa8-30a6-47ef-8766-1efb36d2d8c4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-bpkrz"
Nov 01 05:13:27.772438 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:27.772392    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bffb1fa8-30a6-47ef-8766-1efb36d2d8c4-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-bpkrz\" (UID: \"bffb1fa8-30a6-47ef-8766-1efb36d2d8c4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-bpkrz"
Nov 01 05:13:27.772521 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:27.772431    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bffb1fa8-30a6-47ef-8766-1efb36d2d8c4-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-bpkrz\" (UID: \"bffb1fa8-30a6-47ef-8766-1efb36d2d8c4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-bpkrz"
Nov 01 05:13:27.774502 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:27.774479    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bffb1fa8-30a6-47ef-8766-1efb36d2d8c4-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-bpkrz\" (UID: \"bffb1fa8-30a6-47ef-8766-1efb36d2d8c4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-bpkrz"
Nov 01 05:13:27.774666 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:27.774648    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bffb1fa8-30a6-47ef-8766-1efb36d2d8c4-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-bpkrz\" (UID: \"bffb1fa8-30a6-47ef-8766-1efb36d2d8c4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-bpkrz"
Nov 01 05:13:27.832898 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:27.832869    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctpq2\" (UniqueName: \"kubernetes.io/projected/bffb1fa8-30a6-47ef-8766-1efb36d2d8c4-kube-api-access-ctpq2\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-bpkrz\" (UID: \"bffb1fa8-30a6-47ef-8766-1efb36d2d8c4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-bpkrz"
Nov 01 05:13:27.851737 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:27.851711    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-bpkrz"
Nov 01 05:13:28.226430 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:28.226347    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-bpkrz"]
Nov 01 05:13:28.230087 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:13:28.230063    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbffb1fa8_30a6_47ef_8766_1efb36d2d8c4.slice/crio-378f984e277f970200033bf6d39452f93127c1aa340d6fe42e77aa4eabcd2620 WatchSource:0}: Error finding container 378f984e277f970200033bf6d39452f93127c1aa340d6fe42e77aa4eabcd2620: Status 404 returned error can't find the container with id 378f984e277f970200033bf6d39452f93127c1aa340d6fe42e77aa4eabcd2620
Nov 01 05:13:28.370463 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:28.370417    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-bpkrz" event={"ID":"bffb1fa8-30a6-47ef-8766-1efb36d2d8c4","Type":"ContainerStarted","Data":"378f984e277f970200033bf6d39452f93127c1aa340d6fe42e77aa4eabcd2620"}
Nov 01 05:13:29.377002 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:29.376963    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-bpkrz" event={"ID":"bffb1fa8-30a6-47ef-8766-1efb36d2d8c4","Type":"ContainerStarted","Data":"3b99bb803e32eb6a57a1090e41299e9075d42512cafa066b5874749f54737e7b"}
Nov 01 05:13:29.494282 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:29.494219    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-bpkrz" podStartSLOduration=1.918374911 podStartE2EDuration="2.494199656s" totalImagesPullingTime="575.824745ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:13:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:13:29.493860606 +0000 UTC m=+587.564657329" watchObservedRunningTime="2026-11-01 05:13:29.494199656 +0000 UTC m=+587.564996377"
Nov 01 05:13:29.851994 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:29.851956    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-bpkrz"
Nov 01 05:13:29.924247 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:29.924215    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:13:30.072372 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:13:30.072336    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:13:30.485470 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:30.485420    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:13:30.485749 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:30.485513    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:13:30.485749 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:30.485542    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:13:31.402415 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:31.402343    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:13:31.409832 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:31.409796    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jxzw" event={"ID":"54667545-c5d3-4893-9ade-e16c82cb5b93","Type":"ContainerStarted","Data":"ff5e79c1f7e800789000628267819a6eceeee654a883f86558a39889f235a2e1"}
Nov 01 05:13:31.600551 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:31.584429    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jxzw" podStartSLOduration=2.6451034289999997 podStartE2EDuration="6.58440943s" totalImagesPullingTime="3.939306001s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:13:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:13:31.583032687 +0000 UTC m=+589.653829408" watchObservedRunningTime="2026-11-01 05:13:31.58440943 +0000 UTC m=+589.655206151"
Nov 01 05:13:32.112776 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:32.112743    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jxzw"
Nov 01 05:13:32.114296 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:32.114267    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-4jxzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Nov 01 05:13:32.114446 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:32.114326    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jxzw" podUID="54667545-c5d3-4893-9ade-e16c82cb5b93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Nov 01 05:13:33.112943 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:33.112903    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-4jxzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Nov 01 05:13:33.113342 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:33.112963    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jxzw" podUID="54667545-c5d3-4893-9ade-e16c82cb5b93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Nov 01 05:13:34.112838 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:34.112798    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-4jxzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Nov 01 05:13:34.113013 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:34.112860    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jxzw" podUID="54667545-c5d3-4893-9ade-e16c82cb5b93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Nov 01 05:13:34.518945 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:34.518914    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-jd2fn"]
Nov 01 05:13:34.522573 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:34.522554    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-jd2fn"
Nov 01 05:13:34.540248 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:34.540223    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z4t8\" (UniqueName: \"kubernetes.io/projected/43840797-117e-46c7-8ad5-6a04151b0b54-kube-api-access-4z4t8\") pod \"echo-simulator-kserve-64b6f8f96c-jd2fn\" (UID: \"43840797-117e-46c7-8ad5-6a04151b0b54\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jd2fn"
Nov 01 05:13:34.540339 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:34.540270    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43840797-117e-46c7-8ad5-6a04151b0b54-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-jd2fn\" (UID: \"43840797-117e-46c7-8ad5-6a04151b0b54\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jd2fn"
Nov 01 05:13:34.540380 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:34.540329    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43840797-117e-46c7-8ad5-6a04151b0b54-home\") pod \"echo-simulator-kserve-64b6f8f96c-jd2fn\" (UID: \"43840797-117e-46c7-8ad5-6a04151b0b54\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jd2fn"
Nov 01 05:13:34.540380 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:34.540364    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43840797-117e-46c7-8ad5-6a04151b0b54-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-jd2fn\" (UID: \"43840797-117e-46c7-8ad5-6a04151b0b54\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jd2fn"
Nov 01 05:13:34.540471 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:34.540388    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43840797-117e-46c7-8ad5-6a04151b0b54-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-jd2fn\" (UID: \"43840797-117e-46c7-8ad5-6a04151b0b54\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jd2fn"
Nov 01 05:13:34.540471 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:34.540408    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43840797-117e-46c7-8ad5-6a04151b0b54-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-jd2fn\" (UID: \"43840797-117e-46c7-8ad5-6a04151b0b54\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jd2fn"
Nov 01 05:13:34.624216 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:34.624181    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\""
Nov 01 05:13:34.626201 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:34.626178    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-jd2fn"]
Nov 01 05:13:34.641534 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:34.641509    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z4t8\" (UniqueName: \"kubernetes.io/projected/43840797-117e-46c7-8ad5-6a04151b0b54-kube-api-access-4z4t8\") pod \"echo-simulator-kserve-64b6f8f96c-jd2fn\" (UID: \"43840797-117e-46c7-8ad5-6a04151b0b54\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jd2fn"
Nov 01 05:13:34.641627 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:34.641565    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43840797-117e-46c7-8ad5-6a04151b0b54-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-jd2fn\" (UID: \"43840797-117e-46c7-8ad5-6a04151b0b54\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jd2fn"
Nov 01 05:13:34.641627 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:34.641593    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43840797-117e-46c7-8ad5-6a04151b0b54-home\") pod \"echo-simulator-kserve-64b6f8f96c-jd2fn\" (UID: \"43840797-117e-46c7-8ad5-6a04151b0b54\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jd2fn"
Nov 01 05:13:34.641627 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:34.641618    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43840797-117e-46c7-8ad5-6a04151b0b54-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-jd2fn\" (UID: \"43840797-117e-46c7-8ad5-6a04151b0b54\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jd2fn"
Nov 01 05:13:34.641756 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:34.641646    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43840797-117e-46c7-8ad5-6a04151b0b54-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-jd2fn\" (UID: \"43840797-117e-46c7-8ad5-6a04151b0b54\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jd2fn"
Nov 01 05:13:34.641756 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:34.641683    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43840797-117e-46c7-8ad5-6a04151b0b54-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-jd2fn\" (UID: \"43840797-117e-46c7-8ad5-6a04151b0b54\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jd2fn"
Nov 01 05:13:34.642068 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:34.642044    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43840797-117e-46c7-8ad5-6a04151b0b54-home\") pod \"echo-simulator-kserve-64b6f8f96c-jd2fn\" (UID: \"43840797-117e-46c7-8ad5-6a04151b0b54\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jd2fn"
Nov 01 05:13:34.642143 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:34.642093    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43840797-117e-46c7-8ad5-6a04151b0b54-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-jd2fn\" (UID: \"43840797-117e-46c7-8ad5-6a04151b0b54\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jd2fn"
Nov 01 05:13:34.642143 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:34.642118    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43840797-117e-46c7-8ad5-6a04151b0b54-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-jd2fn\" (UID: \"43840797-117e-46c7-8ad5-6a04151b0b54\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jd2fn"
Nov 01 05:13:34.644091 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:34.644071    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43840797-117e-46c7-8ad5-6a04151b0b54-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-jd2fn\" (UID: \"43840797-117e-46c7-8ad5-6a04151b0b54\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jd2fn"
Nov 01 05:13:34.644378 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:34.644357    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43840797-117e-46c7-8ad5-6a04151b0b54-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-jd2fn\" (UID: \"43840797-117e-46c7-8ad5-6a04151b0b54\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jd2fn"
Nov 01 05:13:34.652811 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:34.652787    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:13:34.699702 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:34.699668    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-d797t"
Nov 01 05:13:34.703119 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:34.703088    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z4t8\" (UniqueName: \"kubernetes.io/projected/43840797-117e-46c7-8ad5-6a04151b0b54-kube-api-access-4z4t8\") pod \"echo-simulator-kserve-64b6f8f96c-jd2fn\" (UID: \"43840797-117e-46c7-8ad5-6a04151b0b54\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jd2fn"
Nov 01 05:13:34.712444 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:34.712421    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-d797t"
Nov 01 05:13:34.835424 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:34.835337    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-jd2fn"
Nov 01 05:13:35.112207 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:35.112129    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-4jxzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Nov 01 05:13:35.112207 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:35.112185    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jxzw" podUID="54667545-c5d3-4893-9ade-e16c82cb5b93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Nov 01 05:13:35.243235 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:35.243205    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-jd2fn"]
Nov 01 05:13:35.246900 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:13:35.246874    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43840797_117e_46c7_8ad5_6a04151b0b54.slice/crio-ddd99a792d0286819bfaa0d497a576ec0a20d31373d3aba3e0064284668449d8 WatchSource:0}: Error finding container ddd99a792d0286819bfaa0d497a576ec0a20d31373d3aba3e0064284668449d8: Status 404 returned error can't find the container with id ddd99a792d0286819bfaa0d497a576ec0a20d31373d3aba3e0064284668449d8
Nov 01 05:13:35.427067 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:35.427035    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-jd2fn" event={"ID":"43840797-117e-46c7-8ad5-6a04151b0b54","Type":"ContainerStarted","Data":"ddd99a792d0286819bfaa0d497a576ec0a20d31373d3aba3e0064284668449d8"}
Nov 01 05:13:35.438376 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:35.438352    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-d797t"
Nov 01 05:13:36.112122 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:36.112093    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-4jxzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Nov 01 05:13:36.112272 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:36.112149    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jxzw" podUID="54667545-c5d3-4893-9ade-e16c82cb5b93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Nov 01 05:13:37.112114 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:37.112080    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-4jxzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Nov 01 05:13:37.112505 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:37.112136    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jxzw" podUID="54667545-c5d3-4893-9ade-e16c82cb5b93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Nov 01 05:13:37.851940 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:37.851915    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-bpkrz"
Nov 01 05:13:37.870032 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:37.869993    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-bpkrz"
Nov 01 05:13:38.443716 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:38.443675    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-jd2fn" event={"ID":"43840797-117e-46c7-8ad5-6a04151b0b54","Type":"ContainerStarted","Data":"05f8faf56625ff365567f411569eaa399a91da1a14c45570237db5c878eb1577"}
Nov 01 05:13:38.455007 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:38.454976    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-bpkrz"
Nov 01 05:13:38.563563 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:38.563509    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-jd2fn" podStartSLOduration=1.98394512 podStartE2EDuration="4.563493372s" totalImagesPullingTime="2.579548252s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:13:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:13:38.561873161 +0000 UTC m=+596.632669884" watchObservedRunningTime="2026-11-01 05:13:38.563493372 +0000 UTC m=+596.634290091"
Nov 01 05:13:38.835853 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:38.835756    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-jd2fn"
Nov 01 05:13:39.153609 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:39.153572    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-4jxzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 05:13:39.153752 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:39.153627    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jxzw" podUID="54667545-c5d3-4893-9ade-e16c82cb5b93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 05:13:39.484661 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:39.484584    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jxzw"
Nov 01 05:13:39.485214 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:39.485195    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jxzw"
Nov 01 05:13:39.486070 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:39.486034    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jxzw"
Nov 01 05:13:40.057952 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:40.057921    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:13:40.616394 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:40.616358    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-2jwpv"]
Nov 01 05:13:40.619922 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:40.619900    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-2jwpv"
Nov 01 05:13:40.695153 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:40.695125    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5944a5e4-357d-4c82-930f-17b1f022f376-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-2jwpv\" (UID: \"5944a5e4-357d-4c82-930f-17b1f022f376\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-2jwpv"
Nov 01 05:13:40.695314 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:40.695165    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5944a5e4-357d-4c82-930f-17b1f022f376-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-2jwpv\" (UID: \"5944a5e4-357d-4c82-930f-17b1f022f376\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-2jwpv"
Nov 01 05:13:40.695314 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:40.695195    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5944a5e4-357d-4c82-930f-17b1f022f376-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-2jwpv\" (UID: \"5944a5e4-357d-4c82-930f-17b1f022f376\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-2jwpv"
Nov 01 05:13:40.695314 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:40.695285    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9dtz\" (UniqueName: \"kubernetes.io/projected/5944a5e4-357d-4c82-930f-17b1f022f376-kube-api-access-q9dtz\") pod \"maas-default-gateway-openshift-default-58d898bd55-2jwpv\" (UID: \"5944a5e4-357d-4c82-930f-17b1f022f376\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-2jwpv"
Nov 01 05:13:40.695417 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:40.695331    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5944a5e4-357d-4c82-930f-17b1f022f376-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-2jwpv\" (UID: \"5944a5e4-357d-4c82-930f-17b1f022f376\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-2jwpv"
Nov 01 05:13:40.695417 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:40.695347    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5944a5e4-357d-4c82-930f-17b1f022f376-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-2jwpv\" (UID: \"5944a5e4-357d-4c82-930f-17b1f022f376\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-2jwpv"
Nov 01 05:13:40.695417 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:40.695392    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5944a5e4-357d-4c82-930f-17b1f022f376-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-2jwpv\" (UID: \"5944a5e4-357d-4c82-930f-17b1f022f376\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-2jwpv"
Nov 01 05:13:40.695534 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:40.695511    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5944a5e4-357d-4c82-930f-17b1f022f376-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-2jwpv\" (UID: \"5944a5e4-357d-4c82-930f-17b1f022f376\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-2jwpv"
Nov 01 05:13:40.695590 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:40.695576    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5944a5e4-357d-4c82-930f-17b1f022f376-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-2jwpv\" (UID: \"5944a5e4-357d-4c82-930f-17b1f022f376\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-2jwpv"
Nov 01 05:13:40.722560 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:40.722528    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-2jwpv"]
Nov 01 05:13:40.796068 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:40.796038    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5944a5e4-357d-4c82-930f-17b1f022f376-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-2jwpv\" (UID: \"5944a5e4-357d-4c82-930f-17b1f022f376\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-2jwpv"
Nov 01 05:13:40.796068 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:40.796069    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5944a5e4-357d-4c82-930f-17b1f022f376-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-2jwpv\" (UID: \"5944a5e4-357d-4c82-930f-17b1f022f376\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-2jwpv"
Nov 01 05:13:40.796266 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:40.796094    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5944a5e4-357d-4c82-930f-17b1f022f376-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-2jwpv\" (UID: \"5944a5e4-357d-4c82-930f-17b1f022f376\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-2jwpv"
Nov 01 05:13:40.796266 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:40.796144    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5944a5e4-357d-4c82-930f-17b1f022f376-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-2jwpv\" (UID: \"5944a5e4-357d-4c82-930f-17b1f022f376\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-2jwpv"
Nov 01 05:13:40.796266 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:40.796171    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5944a5e4-357d-4c82-930f-17b1f022f376-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-2jwpv\" (UID: \"5944a5e4-357d-4c82-930f-17b1f022f376\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-2jwpv"
Nov 01 05:13:40.796266 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:40.796191    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5944a5e4-357d-4c82-930f-17b1f022f376-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-2jwpv\" (UID: \"5944a5e4-357d-4c82-930f-17b1f022f376\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-2jwpv"
Nov 01 05:13:40.796266 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:40.796218    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9dtz\" (UniqueName: \"kubernetes.io/projected/5944a5e4-357d-4c82-930f-17b1f022f376-kube-api-access-q9dtz\") pod \"maas-default-gateway-openshift-default-58d898bd55-2jwpv\" (UID: \"5944a5e4-357d-4c82-930f-17b1f022f376\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-2jwpv"
Nov 01 05:13:40.796541 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:40.796283    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5944a5e4-357d-4c82-930f-17b1f022f376-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-2jwpv\" (UID: \"5944a5e4-357d-4c82-930f-17b1f022f376\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-2jwpv"
Nov 01 05:13:40.796602 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:40.796545    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5944a5e4-357d-4c82-930f-17b1f022f376-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-2jwpv\" (UID: \"5944a5e4-357d-4c82-930f-17b1f022f376\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-2jwpv"
Nov 01 05:13:40.796717 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:40.796696    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5944a5e4-357d-4c82-930f-17b1f022f376-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-2jwpv\" (UID: \"5944a5e4-357d-4c82-930f-17b1f022f376\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-2jwpv"
Nov 01 05:13:40.796811 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:40.796714    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5944a5e4-357d-4c82-930f-17b1f022f376-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-2jwpv\" (UID: \"5944a5e4-357d-4c82-930f-17b1f022f376\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-2jwpv"
Nov 01 05:13:40.796864 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:40.796847    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5944a5e4-357d-4c82-930f-17b1f022f376-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-2jwpv\" (UID: \"5944a5e4-357d-4c82-930f-17b1f022f376\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-2jwpv"
Nov 01 05:13:40.797048 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:40.797016    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5944a5e4-357d-4c82-930f-17b1f022f376-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-2jwpv\" (UID: \"5944a5e4-357d-4c82-930f-17b1f022f376\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-2jwpv"
Nov 01 05:13:40.797341 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:40.797325    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5944a5e4-357d-4c82-930f-17b1f022f376-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-2jwpv\" (UID: \"5944a5e4-357d-4c82-930f-17b1f022f376\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-2jwpv"
Nov 01 05:13:40.798746 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:40.798726    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5944a5e4-357d-4c82-930f-17b1f022f376-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-2jwpv\" (UID: \"5944a5e4-357d-4c82-930f-17b1f022f376\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-2jwpv"
Nov 01 05:13:40.798925 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:40.798907    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5944a5e4-357d-4c82-930f-17b1f022f376-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-2jwpv\" (UID: \"5944a5e4-357d-4c82-930f-17b1f022f376\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-2jwpv"
Nov 01 05:13:40.857128 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:40.857098    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5944a5e4-357d-4c82-930f-17b1f022f376-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-2jwpv\" (UID: \"5944a5e4-357d-4c82-930f-17b1f022f376\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-2jwpv"
Nov 01 05:13:40.857273 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:40.857255    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9dtz\" (UniqueName: \"kubernetes.io/projected/5944a5e4-357d-4c82-930f-17b1f022f376-kube-api-access-q9dtz\") pod \"maas-default-gateway-openshift-default-58d898bd55-2jwpv\" (UID: \"5944a5e4-357d-4c82-930f-17b1f022f376\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-2jwpv"
Nov 01 05:13:40.933293 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:40.933267    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-2jwpv"
Nov 01 05:13:41.359320 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:41.359243    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-2jwpv"]
Nov 01 05:13:41.362378 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:13:41.362350    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5944a5e4_357d_4c82_930f_17b1f022f376.slice/crio-08776ae7307041f825b1e091e6fd4d16ae8460387f597505a8cf4b7f1417c710 WatchSource:0}: Error finding container 08776ae7307041f825b1e091e6fd4d16ae8460387f597505a8cf4b7f1417c710: Status 404 returned error can't find the container with id 08776ae7307041f825b1e091e6fd4d16ae8460387f597505a8cf4b7f1417c710
Nov 01 05:13:41.365507 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:41.365473    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:13:41.365569 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:41.365545    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:13:41.365604 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:41.365573    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:13:41.460318 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:41.460283    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-2jwpv" event={"ID":"5944a5e4-357d-4c82-930f-17b1f022f376","Type":"ContainerStarted","Data":"472eba9828ecd5b8ab3b41248e35904b9176350ceab42b5d8ccc6207f5f2c50c"}
Nov 01 05:13:41.460483 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:41.460346    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-2jwpv" event={"ID":"5944a5e4-357d-4c82-930f-17b1f022f376","Type":"ContainerStarted","Data":"08776ae7307041f825b1e091e6fd4d16ae8460387f597505a8cf4b7f1417c710"}
Nov 01 05:13:41.535185 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:41.535136    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:13:41.580504 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:41.580431    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-2jwpv" podStartSLOduration=1.5804110919999999 podStartE2EDuration="1.580411092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:13:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:13:41.579707439 +0000 UTC m=+599.650504161" watchObservedRunningTime="2026-11-01 05:13:41.580411092 +0000 UTC m=+599.651207806"
Nov 01 05:13:41.712164 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:41.712135    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:13:41.934367 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:41.934332    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-2jwpv"
Nov 01 05:13:42.934625 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:42.934589    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-2jwpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 01 05:13:42.934991 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:42.934641    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-2jwpv" podUID="5944a5e4-357d-4c82-930f-17b1f022f376" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 01 05:13:43.934612 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:43.934575    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-2jwpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 01 05:13:43.934761 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:43.934646    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-2jwpv" podUID="5944a5e4-357d-4c82-930f-17b1f022f376" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 01 05:13:44.835476 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:44.835417    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-jd2fn"
Nov 01 05:13:44.847963 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:44.847939    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-jd2fn"
Nov 01 05:13:44.933658 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:44.933629    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-2jwpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 01 05:13:44.933815 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:44.933688    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-2jwpv" podUID="5944a5e4-357d-4c82-930f-17b1f022f376" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 01 05:13:45.337976 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:45.337947    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:13:45.510519 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:45.510482    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-jd2fn"
Nov 01 05:13:45.721376 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:45.721343    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:13:45.933826 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:45.933774    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-2jwpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 01 05:13:45.934019 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:45.933862    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-2jwpv" podUID="5944a5e4-357d-4c82-930f-17b1f022f376" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 01 05:13:46.933973 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:46.933926    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-2jwpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 01 05:13:46.934528 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:46.933988    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-2jwpv" podUID="5944a5e4-357d-4c82-930f-17b1f022f376" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 01 05:13:47.934354 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:47.934317    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-2jwpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 01 05:13:47.934837 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:47.934379    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-2jwpv" podUID="5944a5e4-357d-4c82-930f-17b1f022f376" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 01 05:13:49.975672 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:49.975625    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-2jwpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 05:13:49.976113 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:49.975702    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-2jwpv" podUID="5944a5e4-357d-4c82-930f-17b1f022f376" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 05:13:50.822411 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:50.822376    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-2jwpv"
Nov 01 05:13:50.822842 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:50.822820    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-2jwpv"
Nov 01 05:13:50.823536 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:50.823515    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-2jwpv"
Nov 01 05:13:55.740542 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:55.740511    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5t46"]
Nov 01 05:13:55.744009 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:55.743992    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5t46"
Nov 01 05:13:55.790092 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:55.790063    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f24gh"]
Nov 01 05:13:55.793503 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:55.793477    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f24gh"
Nov 01 05:13:55.845792 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:55.845763    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/231ce25c-5629-4780-9607-e5cd270afba5-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-h5t46\" (UID: \"231ce25c-5629-4780-9607-e5cd270afba5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5t46"
Nov 01 05:13:55.845918 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:55.845798    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/231ce25c-5629-4780-9607-e5cd270afba5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-h5t46\" (UID: \"231ce25c-5629-4780-9607-e5cd270afba5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5t46"
Nov 01 05:13:55.845918 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:55.845823    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7758bc0a-0b7f-45d6-b688-53936075c3c7-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-f24gh\" (UID: \"7758bc0a-0b7f-45d6-b688-53936075c3c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f24gh"
Nov 01 05:13:55.845918 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:55.845879    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7758bc0a-0b7f-45d6-b688-53936075c3c7-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-f24gh\" (UID: \"7758bc0a-0b7f-45d6-b688-53936075c3c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f24gh"
Nov 01 05:13:55.846085 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:55.845957    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/231ce25c-5629-4780-9607-e5cd270afba5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-h5t46\" (UID: \"231ce25c-5629-4780-9607-e5cd270afba5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5t46"
Nov 01 05:13:55.846085 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:55.846018    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/231ce25c-5629-4780-9607-e5cd270afba5-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-h5t46\" (UID: \"231ce25c-5629-4780-9607-e5cd270afba5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5t46"
Nov 01 05:13:55.846174 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:55.846149    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7758bc0a-0b7f-45d6-b688-53936075c3c7-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-f24gh\" (UID: \"7758bc0a-0b7f-45d6-b688-53936075c3c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f24gh"
Nov 01 05:13:55.846230 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:55.846209    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvv4b\" (UniqueName: \"kubernetes.io/projected/231ce25c-5629-4780-9607-e5cd270afba5-kube-api-access-zvv4b\") pod \"maas-default-gateway-openshift-default-58d898bd55-h5t46\" (UID: \"231ce25c-5629-4780-9607-e5cd270afba5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5t46"
Nov 01 05:13:55.846283 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:55.846255    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/231ce25c-5629-4780-9607-e5cd270afba5-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-h5t46\" (UID: \"231ce25c-5629-4780-9607-e5cd270afba5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5t46"
Nov 01 05:13:55.846347 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:55.846326    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfmbf\" (UniqueName: \"kubernetes.io/projected/7758bc0a-0b7f-45d6-b688-53936075c3c7-kube-api-access-qfmbf\") pod \"maas-default-gateway-openshift-default-58d898bd55-f24gh\" (UID: \"7758bc0a-0b7f-45d6-b688-53936075c3c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f24gh"
Nov 01 05:13:55.846469 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:55.846435    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7758bc0a-0b7f-45d6-b688-53936075c3c7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-f24gh\" (UID: \"7758bc0a-0b7f-45d6-b688-53936075c3c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f24gh"
Nov 01 05:13:55.846526 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:55.846485    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7758bc0a-0b7f-45d6-b688-53936075c3c7-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-f24gh\" (UID: \"7758bc0a-0b7f-45d6-b688-53936075c3c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f24gh"
Nov 01 05:13:55.846526 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:55.846504    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/231ce25c-5629-4780-9607-e5cd270afba5-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-h5t46\" (UID: \"231ce25c-5629-4780-9607-e5cd270afba5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5t46"
Nov 01 05:13:55.846652 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:55.846621    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7758bc0a-0b7f-45d6-b688-53936075c3c7-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-f24gh\" (UID: \"7758bc0a-0b7f-45d6-b688-53936075c3c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f24gh"
Nov 01 05:13:55.846729 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:55.846713    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/231ce25c-5629-4780-9607-e5cd270afba5-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-h5t46\" (UID: \"231ce25c-5629-4780-9607-e5cd270afba5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5t46"
Nov 01 05:13:55.846786 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:55.846746    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/231ce25c-5629-4780-9607-e5cd270afba5-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-h5t46\" (UID: \"231ce25c-5629-4780-9607-e5cd270afba5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5t46"
Nov 01 05:13:55.846840 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:55.846780    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7758bc0a-0b7f-45d6-b688-53936075c3c7-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-f24gh\" (UID: \"7758bc0a-0b7f-45d6-b688-53936075c3c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f24gh"
Nov 01 05:13:55.846840 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:55.846813    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7758bc0a-0b7f-45d6-b688-53936075c3c7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-f24gh\" (UID: \"7758bc0a-0b7f-45d6-b688-53936075c3c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f24gh"
Nov 01 05:13:55.847824 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:55.847803    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5t46"]
Nov 01 05:13:55.857420 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:55.857389    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f24gh"]
Nov 01 05:13:55.947444 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:55.947413    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7758bc0a-0b7f-45d6-b688-53936075c3c7-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-f24gh\" (UID: \"7758bc0a-0b7f-45d6-b688-53936075c3c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f24gh"
Nov 01 05:13:55.947624 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:55.947495    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/231ce25c-5629-4780-9607-e5cd270afba5-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-h5t46\" (UID: \"231ce25c-5629-4780-9607-e5cd270afba5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5t46"
Nov 01 05:13:55.947624 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:55.947516    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/231ce25c-5629-4780-9607-e5cd270afba5-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-h5t46\" (UID: \"231ce25c-5629-4780-9607-e5cd270afba5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5t46"
Nov 01 05:13:55.947624 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:55.947543    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7758bc0a-0b7f-45d6-b688-53936075c3c7-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-f24gh\" (UID: \"7758bc0a-0b7f-45d6-b688-53936075c3c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f24gh"
Nov 01 05:13:55.947624 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:55.947566    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7758bc0a-0b7f-45d6-b688-53936075c3c7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-f24gh\" (UID: \"7758bc0a-0b7f-45d6-b688-53936075c3c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f24gh"
Nov 01 05:13:55.947624 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:55.947601    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/231ce25c-5629-4780-9607-e5cd270afba5-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-h5t46\" (UID: \"231ce25c-5629-4780-9607-e5cd270afba5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5t46"
Nov 01 05:13:55.947885 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:55.947641    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/231ce25c-5629-4780-9607-e5cd270afba5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-h5t46\" (UID: \"231ce25c-5629-4780-9607-e5cd270afba5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5t46"
Nov 01 05:13:55.947885 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:55.947678    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7758bc0a-0b7f-45d6-b688-53936075c3c7-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-f24gh\" (UID: \"7758bc0a-0b7f-45d6-b688-53936075c3c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f24gh"
Nov 01 05:13:55.947885 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:55.947703    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7758bc0a-0b7f-45d6-b688-53936075c3c7-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-f24gh\" (UID: \"7758bc0a-0b7f-45d6-b688-53936075c3c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f24gh"
Nov 01 05:13:55.947885 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:55.947726    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/231ce25c-5629-4780-9607-e5cd270afba5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-h5t46\" (UID: \"231ce25c-5629-4780-9607-e5cd270afba5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5t46"
Nov 01 05:13:55.947885 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:55.947749    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/231ce25c-5629-4780-9607-e5cd270afba5-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-h5t46\" (UID: \"231ce25c-5629-4780-9607-e5cd270afba5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5t46"
Nov 01 05:13:55.947885 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:55.947791    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7758bc0a-0b7f-45d6-b688-53936075c3c7-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-f24gh\" (UID: \"7758bc0a-0b7f-45d6-b688-53936075c3c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f24gh"
Nov 01 05:13:55.947885 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:55.947820    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvv4b\" (UniqueName: \"kubernetes.io/projected/231ce25c-5629-4780-9607-e5cd270afba5-kube-api-access-zvv4b\") pod \"maas-default-gateway-openshift-default-58d898bd55-h5t46\" (UID: \"231ce25c-5629-4780-9607-e5cd270afba5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5t46"
Nov 01 05:13:55.947885 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:55.947854    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/231ce25c-5629-4780-9607-e5cd270afba5-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-h5t46\" (UID: \"231ce25c-5629-4780-9607-e5cd270afba5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5t46"
Nov 01 05:13:55.947885 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:55.947868    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7758bc0a-0b7f-45d6-b688-53936075c3c7-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-f24gh\" (UID: \"7758bc0a-0b7f-45d6-b688-53936075c3c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f24gh"
Nov 01 05:13:55.948507 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:55.947892    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfmbf\" (UniqueName: \"kubernetes.io/projected/7758bc0a-0b7f-45d6-b688-53936075c3c7-kube-api-access-qfmbf\") pod \"maas-default-gateway-openshift-default-58d898bd55-f24gh\" (UID: \"7758bc0a-0b7f-45d6-b688-53936075c3c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f24gh"
Nov 01 05:13:55.948507 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:55.947930    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7758bc0a-0b7f-45d6-b688-53936075c3c7-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-f24gh\" (UID: \"7758bc0a-0b7f-45d6-b688-53936075c3c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f24gh"
Nov 01 05:13:55.948507 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:55.947957    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7758bc0a-0b7f-45d6-b688-53936075c3c7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-f24gh\" (UID: \"7758bc0a-0b7f-45d6-b688-53936075c3c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f24gh"
Nov 01 05:13:55.948507 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:55.947993    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7758bc0a-0b7f-45d6-b688-53936075c3c7-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-f24gh\" (UID: \"7758bc0a-0b7f-45d6-b688-53936075c3c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f24gh"
Nov 01 05:13:55.948507 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:55.948011    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/231ce25c-5629-4780-9607-e5cd270afba5-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-h5t46\" (UID: \"231ce25c-5629-4780-9607-e5cd270afba5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5t46"
Nov 01 05:13:55.948507 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:55.948027    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/231ce25c-5629-4780-9607-e5cd270afba5-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-h5t46\" (UID: \"231ce25c-5629-4780-9607-e5cd270afba5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5t46"
Nov 01 05:13:55.948507 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:55.948069    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/231ce25c-5629-4780-9607-e5cd270afba5-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-h5t46\" (UID: \"231ce25c-5629-4780-9607-e5cd270afba5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5t46"
Nov 01 05:13:55.948507 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:55.948143    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/231ce25c-5629-4780-9607-e5cd270afba5-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-h5t46\" (UID: \"231ce25c-5629-4780-9607-e5cd270afba5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5t46"
Nov 01 05:13:55.948507 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:55.948187    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7758bc0a-0b7f-45d6-b688-53936075c3c7-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-f24gh\" (UID: \"7758bc0a-0b7f-45d6-b688-53936075c3c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f24gh"
Nov 01 05:13:55.948507 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:55.948336    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/231ce25c-5629-4780-9607-e5cd270afba5-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-h5t46\" (UID: \"231ce25c-5629-4780-9607-e5cd270afba5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5t46"
Nov 01 05:13:55.948507 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:55.948334    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7758bc0a-0b7f-45d6-b688-53936075c3c7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-f24gh\" (UID: \"7758bc0a-0b7f-45d6-b688-53936075c3c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f24gh"
Nov 01 05:13:55.948507 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:55.948428    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7758bc0a-0b7f-45d6-b688-53936075c3c7-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-f24gh\" (UID: \"7758bc0a-0b7f-45d6-b688-53936075c3c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f24gh"
Nov 01 05:13:55.949125 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:55.948801    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/231ce25c-5629-4780-9607-e5cd270afba5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-h5t46\" (UID: \"231ce25c-5629-4780-9607-e5cd270afba5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5t46"
Nov 01 05:13:55.950168 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:55.950139    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/231ce25c-5629-4780-9607-e5cd270afba5-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-h5t46\" (UID: \"231ce25c-5629-4780-9607-e5cd270afba5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5t46"
Nov 01 05:13:55.950581 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:55.950556    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7758bc0a-0b7f-45d6-b688-53936075c3c7-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-f24gh\" (UID: \"7758bc0a-0b7f-45d6-b688-53936075c3c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f24gh"
Nov 01 05:13:55.950691 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:55.950663    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/231ce25c-5629-4780-9607-e5cd270afba5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-h5t46\" (UID: \"231ce25c-5629-4780-9607-e5cd270afba5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5t46"
Nov 01 05:13:55.950807 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:55.950789    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7758bc0a-0b7f-45d6-b688-53936075c3c7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-f24gh\" (UID: \"7758bc0a-0b7f-45d6-b688-53936075c3c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f24gh"
Nov 01 05:13:56.014733 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:56.014655    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7758bc0a-0b7f-45d6-b688-53936075c3c7-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-f24gh\" (UID: \"7758bc0a-0b7f-45d6-b688-53936075c3c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f24gh"
Nov 01 05:13:56.014733 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:56.014665    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/231ce25c-5629-4780-9607-e5cd270afba5-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-h5t46\" (UID: \"231ce25c-5629-4780-9607-e5cd270afba5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5t46"
Nov 01 05:13:56.014931 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:56.014774    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfmbf\" (UniqueName: \"kubernetes.io/projected/7758bc0a-0b7f-45d6-b688-53936075c3c7-kube-api-access-qfmbf\") pod \"maas-default-gateway-openshift-default-58d898bd55-f24gh\" (UID: \"7758bc0a-0b7f-45d6-b688-53936075c3c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f24gh"
Nov 01 05:13:56.014931 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:56.014808    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvv4b\" (UniqueName: \"kubernetes.io/projected/231ce25c-5629-4780-9607-e5cd270afba5-kube-api-access-zvv4b\") pod \"maas-default-gateway-openshift-default-58d898bd55-h5t46\" (UID: \"231ce25c-5629-4780-9607-e5cd270afba5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5t46"
Nov 01 05:13:56.056804 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:56.056779    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5t46"
Nov 01 05:13:56.106028 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:56.106002    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f24gh"
Nov 01 05:13:56.482692 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:56.482671    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5t46"]
Nov 01 05:13:56.485264 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:13:56.485242    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod231ce25c_5629_4780_9607_e5cd270afba5.slice/crio-85671cd005352786bd17e184fbc1b6d314cb4b7233ddbc5acee2b099eba069ab WatchSource:0}: Error finding container 85671cd005352786bd17e184fbc1b6d314cb4b7233ddbc5acee2b099eba069ab: Status 404 returned error can't find the container with id 85671cd005352786bd17e184fbc1b6d314cb4b7233ddbc5acee2b099eba069ab
Nov 01 05:13:56.487504 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:56.487474    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:13:56.487565 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:56.487538    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:13:56.487601 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:56.487566    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:13:56.538734 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:56.538711    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f24gh"]
Nov 01 05:13:56.541890 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:13:56.541867    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7758bc0a_0b7f_45d6_b688_53936075c3c7.slice/crio-a644972937021369bebef3028d6ee8eb52d17090397ba953106d7ac4a7afe3b6 WatchSource:0}: Error finding container a644972937021369bebef3028d6ee8eb52d17090397ba953106d7ac4a7afe3b6: Status 404 returned error can't find the container with id a644972937021369bebef3028d6ee8eb52d17090397ba953106d7ac4a7afe3b6
Nov 01 05:13:56.544359 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:56.544324    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:13:56.544469 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:56.544434    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:13:56.544520 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:56.544504    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:13:56.559409 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:56.559378    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5t46" event={"ID":"231ce25c-5629-4780-9607-e5cd270afba5","Type":"ContainerStarted","Data":"96c9d077aea787fd04dfd73dbba04c445ea921469b1f822a5db75386642ee5a0"}
Nov 01 05:13:56.559559 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:56.559473    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5t46" event={"ID":"231ce25c-5629-4780-9607-e5cd270afba5","Type":"ContainerStarted","Data":"85671cd005352786bd17e184fbc1b6d314cb4b7233ddbc5acee2b099eba069ab"}
Nov 01 05:13:56.561106 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:56.561082    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f24gh" event={"ID":"7758bc0a-0b7f-45d6-b688-53936075c3c7","Type":"ContainerStarted","Data":"a644972937021369bebef3028d6ee8eb52d17090397ba953106d7ac4a7afe3b6"}
Nov 01 05:13:56.679982 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:56.679901    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5t46" podStartSLOduration=1.679884343 podStartE2EDuration="1.679884343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:13:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:13:56.678688455 +0000 UTC m=+614.749485175" watchObservedRunningTime="2026-11-01 05:13:56.679884343 +0000 UTC m=+614.750681064"
Nov 01 05:13:56.895820 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:56.895743    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:13:57.057726 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:57.057684    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5t46"
Nov 01 05:13:57.059766 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:57.059735    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-h5t46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Nov 01 05:13:57.059985 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:57.059959    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5t46" podUID="231ce25c-5629-4780-9607-e5cd270afba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Nov 01 05:13:57.425642 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:57.419857    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:13:57.568428 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:57.568396    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f24gh" event={"ID":"7758bc0a-0b7f-45d6-b688-53936075c3c7","Type":"ContainerStarted","Data":"05ba11e9854e456be39666e580103631670cdec4b09a850055b05a8a5f21d0ee"}
Nov 01 05:13:57.701542 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:57.701283    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f24gh" podStartSLOduration=2.70126248 podStartE2EDuration="2.70126248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:13:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:13:57.700763659 +0000 UTC m=+615.771560382" watchObservedRunningTime="2026-11-01 05:13:57.70126248 +0000 UTC m=+615.772059202"
Nov 01 05:13:58.057854 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:58.057774    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-h5t46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Nov 01 05:13:58.057854 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:58.057835    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5t46" podUID="231ce25c-5629-4780-9607-e5cd270afba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Nov 01 05:13:58.118479 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:58.109711    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f24gh"
Nov 01 05:13:58.118479 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:58.113562    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-f24gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Nov 01 05:13:58.118479 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:58.113618    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f24gh" podUID="7758bc0a-0b7f-45d6-b688-53936075c3c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Nov 01 05:13:59.058007 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:59.057975    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-h5t46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Nov 01 05:13:59.058388 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:59.058020    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5t46" podUID="231ce25c-5629-4780-9607-e5cd270afba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Nov 01 05:13:59.111476 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:59.110531    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-f24gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Nov 01 05:13:59.111476 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:13:59.110589    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f24gh" podUID="7758bc0a-0b7f-45d6-b688-53936075c3c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Nov 01 05:14:00.057766 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:00.057730    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-h5t46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Nov 01 05:14:00.057950 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:00.057795    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5t46" podUID="231ce25c-5629-4780-9607-e5cd270afba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Nov 01 05:14:00.068869 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:14:00.068833    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:14:00.107927 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:00.107807    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-f24gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Nov 01 05:14:00.107927 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:00.107869    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f24gh" podUID="7758bc0a-0b7f-45d6-b688-53936075c3c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Nov 01 05:14:01.057797 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:01.057759    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-h5t46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Nov 01 05:14:01.057960 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:01.057815    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5t46" podUID="231ce25c-5629-4780-9607-e5cd270afba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Nov 01 05:14:01.110477 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:01.109789    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-f24gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Nov 01 05:14:01.110477 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:01.109850    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f24gh" podUID="7758bc0a-0b7f-45d6-b688-53936075c3c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Nov 01 05:14:02.057891 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:02.057854    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-h5t46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Nov 01 05:14:02.058073 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:02.057906    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5t46" podUID="231ce25c-5629-4780-9607-e5cd270afba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Nov 01 05:14:02.106912 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:02.106870    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-f24gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Nov 01 05:14:02.107183 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:02.107153    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f24gh" podUID="7758bc0a-0b7f-45d6-b688-53936075c3c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Nov 01 05:14:03.057842 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:03.057800    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-h5t46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Nov 01 05:14:03.058334 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:03.057862    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5t46" podUID="231ce25c-5629-4780-9607-e5cd270afba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Nov 01 05:14:03.107065 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:03.107030    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-f24gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Nov 01 05:14:03.107214 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:03.107075    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f24gh" podUID="7758bc0a-0b7f-45d6-b688-53936075c3c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Nov 01 05:14:04.057408 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:04.057370    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-h5t46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Nov 01 05:14:04.057635 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:04.057428    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5t46" podUID="231ce25c-5629-4780-9607-e5cd270afba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Nov 01 05:14:04.837537 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:04.837505    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f24gh"
Nov 01 05:14:04.838032 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:04.838012    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f24gh"
Nov 01 05:14:04.838806 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:04.838769    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f24gh"
Nov 01 05:14:05.060786 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:05.060751    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5t46"
Nov 01 05:14:05.061102 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:05.061075    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5t46"
Nov 01 05:14:05.061806 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:05.061780    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5t46"
Nov 01 05:14:11.528676 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:11.528639    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:14:11.715058 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:11.715004    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:14:13.131037 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:13.131005    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:14:13.311339 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:13.311303    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:14:30.068694 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:14:30.068616    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:14:50.722561 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:50.722528    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:14:50.903529 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:50.903500    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:14:51.368220 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:51.368173    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:14:51.546677 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:51.546634    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb"]
Nov 01 05:14:51.551384 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:51.551359    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb"
Nov 01 05:14:51.558334 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:51.558107    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:14:51.621820 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:51.621713    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-7d1ec3-openshift-default-dockercfg-thhxl\""
Nov 01 05:14:51.672822 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:51.672785    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb"]
Nov 01 05:14:51.690390 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:51.690356    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8a582fb-747c-4c55-a0e3-b5f5139fa28f-credential-socket\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb\" (UID: \"c8a582fb-747c-4c55-a0e3-b5f5139fa28f\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb"
Nov 01 05:14:51.690555 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:51.690397    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8a582fb-747c-4c55-a0e3-b5f5139fa28f-istio-podinfo\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb\" (UID: \"c8a582fb-747c-4c55-a0e3-b5f5139fa28f\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb"
Nov 01 05:14:51.690555 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:51.690427    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8a582fb-747c-4c55-a0e3-b5f5139fa28f-istio-data\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb\" (UID: \"c8a582fb-747c-4c55-a0e3-b5f5139fa28f\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb"
Nov 01 05:14:51.690670 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:51.690595    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8a582fb-747c-4c55-a0e3-b5f5139fa28f-workload-socket\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb\" (UID: \"c8a582fb-747c-4c55-a0e3-b5f5139fa28f\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb"
Nov 01 05:14:51.690727 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:51.690705    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m9xq\" (UniqueName: \"kubernetes.io/projected/c8a582fb-747c-4c55-a0e3-b5f5139fa28f-kube-api-access-9m9xq\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb\" (UID: \"c8a582fb-747c-4c55-a0e3-b5f5139fa28f\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb"
Nov 01 05:14:51.690779 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:51.690757    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8a582fb-747c-4c55-a0e3-b5f5139fa28f-istio-token\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb\" (UID: \"c8a582fb-747c-4c55-a0e3-b5f5139fa28f\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb"
Nov 01 05:14:51.690833 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:51.690822    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8a582fb-747c-4c55-a0e3-b5f5139fa28f-istiod-ca-cert\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb\" (UID: \"c8a582fb-747c-4c55-a0e3-b5f5139fa28f\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb"
Nov 01 05:14:51.690921 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:51.690904    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8a582fb-747c-4c55-a0e3-b5f5139fa28f-istio-envoy\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb\" (UID: \"c8a582fb-747c-4c55-a0e3-b5f5139fa28f\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb"
Nov 01 05:14:51.690955 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:51.690933    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8a582fb-747c-4c55-a0e3-b5f5139fa28f-workload-certs\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb\" (UID: \"c8a582fb-747c-4c55-a0e3-b5f5139fa28f\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb"
Nov 01 05:14:51.734719 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:51.734686    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:14:51.792337 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:51.792301    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8a582fb-747c-4c55-a0e3-b5f5139fa28f-istio-token\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb\" (UID: \"c8a582fb-747c-4c55-a0e3-b5f5139fa28f\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb"
Nov 01 05:14:51.792509 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:51.792369    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8a582fb-747c-4c55-a0e3-b5f5139fa28f-istiod-ca-cert\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb\" (UID: \"c8a582fb-747c-4c55-a0e3-b5f5139fa28f\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb"
Nov 01 05:14:51.792509 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:51.792422    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8a582fb-747c-4c55-a0e3-b5f5139fa28f-istio-envoy\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb\" (UID: \"c8a582fb-747c-4c55-a0e3-b5f5139fa28f\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb"
Nov 01 05:14:51.792509 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:51.792486    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8a582fb-747c-4c55-a0e3-b5f5139fa28f-workload-certs\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb\" (UID: \"c8a582fb-747c-4c55-a0e3-b5f5139fa28f\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb"
Nov 01 05:14:51.792647 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:51.792549    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8a582fb-747c-4c55-a0e3-b5f5139fa28f-credential-socket\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb\" (UID: \"c8a582fb-747c-4c55-a0e3-b5f5139fa28f\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb"
Nov 01 05:14:51.792647 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:51.792575    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8a582fb-747c-4c55-a0e3-b5f5139fa28f-istio-podinfo\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb\" (UID: \"c8a582fb-747c-4c55-a0e3-b5f5139fa28f\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb"
Nov 01 05:14:51.792647 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:51.792603    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8a582fb-747c-4c55-a0e3-b5f5139fa28f-istio-data\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb\" (UID: \"c8a582fb-747c-4c55-a0e3-b5f5139fa28f\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb"
Nov 01 05:14:51.792647 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:51.792638    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8a582fb-747c-4c55-a0e3-b5f5139fa28f-workload-socket\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb\" (UID: \"c8a582fb-747c-4c55-a0e3-b5f5139fa28f\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb"
Nov 01 05:14:51.792792 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:51.792673    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m9xq\" (UniqueName: \"kubernetes.io/projected/c8a582fb-747c-4c55-a0e3-b5f5139fa28f-kube-api-access-9m9xq\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb\" (UID: \"c8a582fb-747c-4c55-a0e3-b5f5139fa28f\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb"
Nov 01 05:14:51.792971 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:51.792932    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8a582fb-747c-4c55-a0e3-b5f5139fa28f-workload-certs\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb\" (UID: \"c8a582fb-747c-4c55-a0e3-b5f5139fa28f\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb"
Nov 01 05:14:51.793127 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:51.793094    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8a582fb-747c-4c55-a0e3-b5f5139fa28f-credential-socket\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb\" (UID: \"c8a582fb-747c-4c55-a0e3-b5f5139fa28f\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb"
Nov 01 05:14:51.793127 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:51.793103    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8a582fb-747c-4c55-a0e3-b5f5139fa28f-workload-socket\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb\" (UID: \"c8a582fb-747c-4c55-a0e3-b5f5139fa28f\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb"
Nov 01 05:14:51.793262 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:51.793203    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8a582fb-747c-4c55-a0e3-b5f5139fa28f-istiod-ca-cert\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb\" (UID: \"c8a582fb-747c-4c55-a0e3-b5f5139fa28f\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb"
Nov 01 05:14:51.793579 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:51.793551    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8a582fb-747c-4c55-a0e3-b5f5139fa28f-istio-data\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb\" (UID: \"c8a582fb-747c-4c55-a0e3-b5f5139fa28f\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb"
Nov 01 05:14:51.795151 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:51.795126    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8a582fb-747c-4c55-a0e3-b5f5139fa28f-istio-envoy\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb\" (UID: \"c8a582fb-747c-4c55-a0e3-b5f5139fa28f\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb"
Nov 01 05:14:51.795410 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:51.795386    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8a582fb-747c-4c55-a0e3-b5f5139fa28f-istio-podinfo\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb\" (UID: \"c8a582fb-747c-4c55-a0e3-b5f5139fa28f\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb"
Nov 01 05:14:51.854947 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:51.854915    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8a582fb-747c-4c55-a0e3-b5f5139fa28f-istio-token\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb\" (UID: \"c8a582fb-747c-4c55-a0e3-b5f5139fa28f\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb"
Nov 01 05:14:51.855130 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:51.855113    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m9xq\" (UniqueName: \"kubernetes.io/projected/c8a582fb-747c-4c55-a0e3-b5f5139fa28f-kube-api-access-9m9xq\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb\" (UID: \"c8a582fb-747c-4c55-a0e3-b5f5139fa28f\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb"
Nov 01 05:14:51.864136 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:51.864115    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb"
Nov 01 05:14:51.871480 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:51.871440    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg"]
Nov 01 05:14:51.875563 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:51.875547    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg"
Nov 01 05:14:51.929026 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:51.928985    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-3d70ff-openshift-default-dockercfg-x8xrx\""
Nov 01 05:14:51.939271 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:51.939189    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg"]
Nov 01 05:14:51.993933 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:51.993899    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba32daa8-34ac-44b9-a090-d4804cdeed41-istio-data\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg\" (UID: \"ba32daa8-34ac-44b9-a090-d4804cdeed41\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg"
Nov 01 05:14:51.994135 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:51.993953    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba32daa8-34ac-44b9-a090-d4804cdeed41-workload-socket\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg\" (UID: \"ba32daa8-34ac-44b9-a090-d4804cdeed41\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg"
Nov 01 05:14:51.994135 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:51.994087    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba32daa8-34ac-44b9-a090-d4804cdeed41-istio-podinfo\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg\" (UID: \"ba32daa8-34ac-44b9-a090-d4804cdeed41\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg"
Nov 01 05:14:51.994135 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:51.994118    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba32daa8-34ac-44b9-a090-d4804cdeed41-istio-token\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg\" (UID: \"ba32daa8-34ac-44b9-a090-d4804cdeed41\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg"
Nov 01 05:14:51.994248 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:51.994205    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nkng\" (UniqueName: \"kubernetes.io/projected/ba32daa8-34ac-44b9-a090-d4804cdeed41-kube-api-access-4nkng\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg\" (UID: \"ba32daa8-34ac-44b9-a090-d4804cdeed41\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg"
Nov 01 05:14:51.994292 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:51.994249    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba32daa8-34ac-44b9-a090-d4804cdeed41-credential-socket\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg\" (UID: \"ba32daa8-34ac-44b9-a090-d4804cdeed41\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg"
Nov 01 05:14:51.994330 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:51.994317    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba32daa8-34ac-44b9-a090-d4804cdeed41-istiod-ca-cert\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg\" (UID: \"ba32daa8-34ac-44b9-a090-d4804cdeed41\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg"
Nov 01 05:14:51.994367 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:51.994343    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba32daa8-34ac-44b9-a090-d4804cdeed41-istio-envoy\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg\" (UID: \"ba32daa8-34ac-44b9-a090-d4804cdeed41\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg"
Nov 01 05:14:51.994411 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:51.994388    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba32daa8-34ac-44b9-a090-d4804cdeed41-workload-certs\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg\" (UID: \"ba32daa8-34ac-44b9-a090-d4804cdeed41\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg"
Nov 01 05:14:51.995778 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:51.995756    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:14:52.095928 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:52.095892    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nkng\" (UniqueName: \"kubernetes.io/projected/ba32daa8-34ac-44b9-a090-d4804cdeed41-kube-api-access-4nkng\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg\" (UID: \"ba32daa8-34ac-44b9-a090-d4804cdeed41\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg"
Nov 01 05:14:52.096099 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:52.095938    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba32daa8-34ac-44b9-a090-d4804cdeed41-credential-socket\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg\" (UID: \"ba32daa8-34ac-44b9-a090-d4804cdeed41\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg"
Nov 01 05:14:52.096099 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:52.095976    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba32daa8-34ac-44b9-a090-d4804cdeed41-istiod-ca-cert\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg\" (UID: \"ba32daa8-34ac-44b9-a090-d4804cdeed41\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg"
Nov 01 05:14:52.096099 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:52.096000    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba32daa8-34ac-44b9-a090-d4804cdeed41-istio-envoy\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg\" (UID: \"ba32daa8-34ac-44b9-a090-d4804cdeed41\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg"
Nov 01 05:14:52.096099 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:52.096036    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba32daa8-34ac-44b9-a090-d4804cdeed41-workload-certs\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg\" (UID: \"ba32daa8-34ac-44b9-a090-d4804cdeed41\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg"
Nov 01 05:14:52.096310 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:52.096138    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba32daa8-34ac-44b9-a090-d4804cdeed41-istio-data\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg\" (UID: \"ba32daa8-34ac-44b9-a090-d4804cdeed41\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg"
Nov 01 05:14:52.096310 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:52.096168    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba32daa8-34ac-44b9-a090-d4804cdeed41-workload-socket\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg\" (UID: \"ba32daa8-34ac-44b9-a090-d4804cdeed41\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg"
Nov 01 05:14:52.096310 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:52.096211    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba32daa8-34ac-44b9-a090-d4804cdeed41-istio-podinfo\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg\" (UID: \"ba32daa8-34ac-44b9-a090-d4804cdeed41\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg"
Nov 01 05:14:52.096496 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:52.096336    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba32daa8-34ac-44b9-a090-d4804cdeed41-istio-token\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg\" (UID: \"ba32daa8-34ac-44b9-a090-d4804cdeed41\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg"
Nov 01 05:14:52.096496 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:52.096382    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba32daa8-34ac-44b9-a090-d4804cdeed41-credential-socket\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg\" (UID: \"ba32daa8-34ac-44b9-a090-d4804cdeed41\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg"
Nov 01 05:14:52.096676 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:52.096627    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba32daa8-34ac-44b9-a090-d4804cdeed41-workload-certs\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg\" (UID: \"ba32daa8-34ac-44b9-a090-d4804cdeed41\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg"
Nov 01 05:14:52.096786 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:52.096741    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba32daa8-34ac-44b9-a090-d4804cdeed41-istio-data\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg\" (UID: \"ba32daa8-34ac-44b9-a090-d4804cdeed41\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg"
Nov 01 05:14:52.096839 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:52.096814    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba32daa8-34ac-44b9-a090-d4804cdeed41-istiod-ca-cert\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg\" (UID: \"ba32daa8-34ac-44b9-a090-d4804cdeed41\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg"
Nov 01 05:14:52.097024 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:52.097000    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba32daa8-34ac-44b9-a090-d4804cdeed41-workload-socket\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg\" (UID: \"ba32daa8-34ac-44b9-a090-d4804cdeed41\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg"
Nov 01 05:14:52.098467 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:52.098438    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba32daa8-34ac-44b9-a090-d4804cdeed41-istio-envoy\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg\" (UID: \"ba32daa8-34ac-44b9-a090-d4804cdeed41\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg"
Nov 01 05:14:52.098767 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:52.098748    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba32daa8-34ac-44b9-a090-d4804cdeed41-istio-podinfo\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg\" (UID: \"ba32daa8-34ac-44b9-a090-d4804cdeed41\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg"
Nov 01 05:14:52.156487 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:52.156441    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba32daa8-34ac-44b9-a090-d4804cdeed41-istio-token\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg\" (UID: \"ba32daa8-34ac-44b9-a090-d4804cdeed41\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg"
Nov 01 05:14:52.156627 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:52.156533    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nkng\" (UniqueName: \"kubernetes.io/projected/ba32daa8-34ac-44b9-a090-d4804cdeed41-kube-api-access-4nkng\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg\" (UID: \"ba32daa8-34ac-44b9-a090-d4804cdeed41\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg"
Nov 01 05:14:52.184355 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:52.184265    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:14:52.194186 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:52.194159    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg"
Nov 01 05:14:52.303161 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:52.303128    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb"]
Nov 01 05:14:52.306269 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:14:52.306235    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8a582fb_747c_4c55_a0e3_b5f5139fa28f.slice/crio-3ae51e7ad3332067af9dec10f8a59e3d4df834f5be5545294a2245513c0caee2 WatchSource:0}: Error finding container 3ae51e7ad3332067af9dec10f8a59e3d4df834f5be5545294a2245513c0caee2: Status 404 returned error can't find the container with id 3ae51e7ad3332067af9dec10f8a59e3d4df834f5be5545294a2245513c0caee2
Nov 01 05:14:52.308561 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:52.308529    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:14:52.308637 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:52.308596    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:14:52.308637 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:52.308633    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:14:52.576117 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:52.576052    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg"]
Nov 01 05:14:52.579383 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:14:52.579355    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba32daa8_34ac_44b9_a090_d4804cdeed41.slice/crio-e68a3fbf6542d90733dc72e32ce38c427a72450ae53cfafcf43a6f0271790bb4 WatchSource:0}: Error finding container e68a3fbf6542d90733dc72e32ce38c427a72450ae53cfafcf43a6f0271790bb4: Status 404 returned error can't find the container with id e68a3fbf6542d90733dc72e32ce38c427a72450ae53cfafcf43a6f0271790bb4
Nov 01 05:14:52.581760 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:52.581730    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:14:52.581846 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:52.581791    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:14:52.581846 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:52.581823    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:14:52.839101 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:52.839016    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb" event={"ID":"c8a582fb-747c-4c55-a0e3-b5f5139fa28f","Type":"ContainerStarted","Data":"7cadc1665174645d53c05fbe46ab5b2ab55f637c9e8ac49e4e497655859406a2"}
Nov 01 05:14:52.839101 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:52.839075    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb" event={"ID":"c8a582fb-747c-4c55-a0e3-b5f5139fa28f","Type":"ContainerStarted","Data":"3ae51e7ad3332067af9dec10f8a59e3d4df834f5be5545294a2245513c0caee2"}
Nov 01 05:14:52.840814 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:52.840786    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg" event={"ID":"ba32daa8-34ac-44b9-a090-d4804cdeed41","Type":"ContainerStarted","Data":"e20fe1cec4efa6ca8c184e93cc6b2de1917576e46459be9b2c80b3b9130645c5"}
Nov 01 05:14:52.840935 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:52.840830    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg" event={"ID":"ba32daa8-34ac-44b9-a090-d4804cdeed41","Type":"ContainerStarted","Data":"e68a3fbf6542d90733dc72e32ce38c427a72450ae53cfafcf43a6f0271790bb4"}
Nov 01 05:14:52.865172 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:52.865146    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb"
Nov 01 05:14:52.960121 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:52.960070    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb" podStartSLOduration=1.960056191 podStartE2EDuration="1.960056191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:14:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:14:52.958920438 +0000 UTC m=+671.029717158" watchObservedRunningTime="2026-11-01 05:14:52.960056191 +0000 UTC m=+671.030852911"
Nov 01 05:14:53.084432 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:53.080444    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg" podStartSLOduration=2.080423055 podStartE2EDuration="2.080423055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:14:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:14:53.079955478 +0000 UTC m=+671.150752224" watchObservedRunningTime="2026-11-01 05:14:53.080423055 +0000 UTC m=+671.151219784"
Nov 01 05:14:53.195010 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:53.194906    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg"
Nov 01 05:14:53.207497 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:53.207227    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg"
Nov 01 05:14:53.845497 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:53.845441    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg"
Nov 01 05:14:53.846406 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:53.846386    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg"
Nov 01 05:14:53.868983 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:53.868962    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb"
Nov 01 05:14:54.850946 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:54.850916    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb"
Nov 01 05:14:54.851941 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:54.851922    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb"
Nov 01 05:14:55.231490 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:55.231445    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:14:55.409833 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:55.409803    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:14:55.592110 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:55.592030    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:14:55.924289 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:55.924256    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:14:56.101978 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:56.101947    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:14:56.364491 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:56.364371    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:14:56.539542 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:56.539483    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:14:56.721877 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:56.721845    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:14:56.902595 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:56.902564    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:14:57.212622 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:57.212592    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:14:57.396362 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:57.396322    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:14:58.807489 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:58.807433    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:14:58.986738 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:14:58.986701    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:15:00.068184 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:15:00.068153    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:15:00.166893 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:00.166853    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891835-h469d"]
Nov 01 05:15:00.170514 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:00.170495    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891835-h469d"
Nov 01 05:15:00.222746 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:00.222716    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\""
Nov 01 05:15:00.273544 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:00.273508    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891835-h469d"]
Nov 01 05:15:00.293307 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:00.293278    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24kbf\" (UniqueName: \"kubernetes.io/projected/02597e35-83cc-42f6-9dea-e1b04942fe9e-kube-api-access-24kbf\") pod \"maas-api-key-cleanup-29891835-h469d\" (UID: \"02597e35-83cc-42f6-9dea-e1b04942fe9e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891835-h469d"
Nov 01 05:15:00.293482 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:00.293330    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02597e35-83cc-42f6-9dea-e1b04942fe9e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891835-h469d\" (UID: \"02597e35-83cc-42f6-9dea-e1b04942fe9e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891835-h469d"
Nov 01 05:15:00.395120 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:00.395022    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24kbf\" (UniqueName: \"kubernetes.io/projected/02597e35-83cc-42f6-9dea-e1b04942fe9e-kube-api-access-24kbf\") pod \"maas-api-key-cleanup-29891835-h469d\" (UID: \"02597e35-83cc-42f6-9dea-e1b04942fe9e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891835-h469d"
Nov 01 05:15:00.395120 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:00.395081    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02597e35-83cc-42f6-9dea-e1b04942fe9e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891835-h469d\" (UID: \"02597e35-83cc-42f6-9dea-e1b04942fe9e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891835-h469d"
Nov 01 05:15:00.395770 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:00.395739    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02597e35-83cc-42f6-9dea-e1b04942fe9e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891835-h469d\" (UID: \"02597e35-83cc-42f6-9dea-e1b04942fe9e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891835-h469d"
Nov 01 05:15:00.456534 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:00.456505    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24kbf\" (UniqueName: \"kubernetes.io/projected/02597e35-83cc-42f6-9dea-e1b04942fe9e-kube-api-access-24kbf\") pod \"maas-api-key-cleanup-29891835-h469d\" (UID: \"02597e35-83cc-42f6-9dea-e1b04942fe9e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891835-h469d"
Nov 01 05:15:00.484072 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:00.484042    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891835-h469d"
Nov 01 05:15:00.877963 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:00.877935    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891835-h469d"]
Nov 01 05:15:00.881207 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:15:00.881172    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02597e35_83cc_42f6_9dea_e1b04942fe9e.slice/crio-bda6b01cdb5664ee7f62f95ae5d75d42be96be5acdebcb88051f32d204f714e3 WatchSource:0}: Error finding container bda6b01cdb5664ee7f62f95ae5d75d42be96be5acdebcb88051f32d204f714e3: Status 404 returned error can't find the container with id bda6b01cdb5664ee7f62f95ae5d75d42be96be5acdebcb88051f32d204f714e3
Nov 01 05:15:00.883246 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:00.883223    5689 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 05:15:01.425105 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:01.425071    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-7d1ec3-55d75cf7c8-25l2f"]
Nov 01 05:15:01.429891 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:01.429868    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7d1ec3-55d75cf7c8-25l2f"
Nov 01 05:15:01.482040 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:01.482005    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-7d1ec3-serving-cert\""
Nov 01 05:15:01.510786 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:01.510754    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4ef14f8-4441-4dba-ace6-fad77c35a0b2-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-7d1ec3-55d75cf7c8-25l2f\" (UID: \"f4ef14f8-4441-4dba-ace6-fad77c35a0b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7d1ec3-55d75cf7c8-25l2f"
Nov 01 05:15:01.510985 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:01.510789    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4ef14f8-4441-4dba-ace6-fad77c35a0b2-metrics-certs\") pod \"maas-api-e2e-worker-w0-7d1ec3-55d75cf7c8-25l2f\" (UID: \"f4ef14f8-4441-4dba-ace6-fad77c35a0b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7d1ec3-55d75cf7c8-25l2f"
Nov 01 05:15:01.510985 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:01.510812    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-729lb\" (UniqueName: \"kubernetes.io/projected/f4ef14f8-4441-4dba-ace6-fad77c35a0b2-kube-api-access-729lb\") pod \"maas-api-e2e-worker-w0-7d1ec3-55d75cf7c8-25l2f\" (UID: \"f4ef14f8-4441-4dba-ace6-fad77c35a0b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7d1ec3-55d75cf7c8-25l2f"
Nov 01 05:15:01.510985 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:01.510842    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4ef14f8-4441-4dba-ace6-fad77c35a0b2-maas-api-tls\") pod \"maas-api-e2e-worker-w0-7d1ec3-55d75cf7c8-25l2f\" (UID: \"f4ef14f8-4441-4dba-ace6-fad77c35a0b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7d1ec3-55d75cf7c8-25l2f"
Nov 01 05:15:01.531581 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:01.531555    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 01 05:15:01.531581 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:01.531557    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9729w\""
Nov 01 05:15:01.533041 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:01.533023    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-7d1ec3-55d75cf7c8-25l2f"]
Nov 01 05:15:01.611602 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:01.611574    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4ef14f8-4441-4dba-ace6-fad77c35a0b2-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-7d1ec3-55d75cf7c8-25l2f\" (UID: \"f4ef14f8-4441-4dba-ace6-fad77c35a0b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7d1ec3-55d75cf7c8-25l2f"
Nov 01 05:15:01.611777 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:01.611610    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4ef14f8-4441-4dba-ace6-fad77c35a0b2-metrics-certs\") pod \"maas-api-e2e-worker-w0-7d1ec3-55d75cf7c8-25l2f\" (UID: \"f4ef14f8-4441-4dba-ace6-fad77c35a0b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7d1ec3-55d75cf7c8-25l2f"
Nov 01 05:15:01.611777 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:01.611630    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-729lb\" (UniqueName: \"kubernetes.io/projected/f4ef14f8-4441-4dba-ace6-fad77c35a0b2-kube-api-access-729lb\") pod \"maas-api-e2e-worker-w0-7d1ec3-55d75cf7c8-25l2f\" (UID: \"f4ef14f8-4441-4dba-ace6-fad77c35a0b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7d1ec3-55d75cf7c8-25l2f"
Nov 01 05:15:01.611777 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:01.611654    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4ef14f8-4441-4dba-ace6-fad77c35a0b2-maas-api-tls\") pod \"maas-api-e2e-worker-w0-7d1ec3-55d75cf7c8-25l2f\" (UID: \"f4ef14f8-4441-4dba-ace6-fad77c35a0b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7d1ec3-55d75cf7c8-25l2f"
Nov 01 05:15:01.611968 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:15:01.611832    5689 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-7d1ec3-serving-cert: secret "maas-api-e2e-worker-w0-7d1ec3-serving-cert" not found
Nov 01 05:15:01.611968 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:15:01.611925    5689 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4ef14f8-4441-4dba-ace6-fad77c35a0b2-maas-api-tls podName:f4ef14f8-4441-4dba-ace6-fad77c35a0b2 nodeName:}" failed. No retries permitted until 2026-11-01 05:15:02.111902653 +0000 UTC m=+680.182699366 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f4ef14f8-4441-4dba-ace6-fad77c35a0b2-maas-api-tls") pod "maas-api-e2e-worker-w0-7d1ec3-55d75cf7c8-25l2f" (UID: "f4ef14f8-4441-4dba-ace6-fad77c35a0b2") : secret "maas-api-e2e-worker-w0-7d1ec3-serving-cert" not found
Nov 01 05:15:01.612270 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:01.612248    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4ef14f8-4441-4dba-ace6-fad77c35a0b2-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-7d1ec3-55d75cf7c8-25l2f\" (UID: \"f4ef14f8-4441-4dba-ace6-fad77c35a0b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7d1ec3-55d75cf7c8-25l2f"
Nov 01 05:15:01.614131 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:01.614114    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4ef14f8-4441-4dba-ace6-fad77c35a0b2-metrics-certs\") pod \"maas-api-e2e-worker-w0-7d1ec3-55d75cf7c8-25l2f\" (UID: \"f4ef14f8-4441-4dba-ace6-fad77c35a0b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7d1ec3-55d75cf7c8-25l2f"
Nov 01 05:15:01.652132 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:01.652107    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:15:01.688939 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:01.688827    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-729lb\" (UniqueName: \"kubernetes.io/projected/f4ef14f8-4441-4dba-ace6-fad77c35a0b2-kube-api-access-729lb\") pod \"maas-api-e2e-worker-w0-7d1ec3-55d75cf7c8-25l2f\" (UID: \"f4ef14f8-4441-4dba-ace6-fad77c35a0b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7d1ec3-55d75cf7c8-25l2f"
Nov 01 05:15:01.834392 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:01.834352    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:15:01.891534 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:01.891495    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891835-h469d" event={"ID":"02597e35-83cc-42f6-9dea-e1b04942fe9e","Type":"ContainerStarted","Data":"bda6b01cdb5664ee7f62f95ae5d75d42be96be5acdebcb88051f32d204f714e3"}
Nov 01 05:15:02.022277 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:02.022204    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:15:02.116799 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:02.116765    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4ef14f8-4441-4dba-ace6-fad77c35a0b2-maas-api-tls\") pod \"maas-api-e2e-worker-w0-7d1ec3-55d75cf7c8-25l2f\" (UID: \"f4ef14f8-4441-4dba-ace6-fad77c35a0b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7d1ec3-55d75cf7c8-25l2f"
Nov 01 05:15:02.119281 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:02.119251    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4ef14f8-4441-4dba-ace6-fad77c35a0b2-maas-api-tls\") pod \"maas-api-e2e-worker-w0-7d1ec3-55d75cf7c8-25l2f\" (UID: \"f4ef14f8-4441-4dba-ace6-fad77c35a0b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7d1ec3-55d75cf7c8-25l2f"
Nov 01 05:15:02.345680 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:02.345597    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7d1ec3-55d75cf7c8-25l2f"
Nov 01 05:15:02.723095 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:02.723067    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-7d1ec3-55d75cf7c8-25l2f"]
Nov 01 05:15:02.726498 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:15:02.726469    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4ef14f8_4441_4dba_ace6_fad77c35a0b2.slice/crio-fdbe47df34c6d88b76283f2a774ab8f5eeb7be693506bf11c4a01840b9934542 WatchSource:0}: Error finding container fdbe47df34c6d88b76283f2a774ab8f5eeb7be693506bf11c4a01840b9934542: Status 404 returned error can't find the container with id fdbe47df34c6d88b76283f2a774ab8f5eeb7be693506bf11c4a01840b9934542
Nov 01 05:15:02.897388 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:02.897356    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7d1ec3-55d75cf7c8-25l2f" event={"ID":"f4ef14f8-4441-4dba-ace6-fad77c35a0b2","Type":"ContainerStarted","Data":"fdbe47df34c6d88b76283f2a774ab8f5eeb7be693506bf11c4a01840b9934542"}
Nov 01 05:15:03.685936 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:03.685905    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:15:03.864732 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:03.864689    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:15:03.904026 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:03.903985    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7d1ec3-55d75cf7c8-25l2f" event={"ID":"f4ef14f8-4441-4dba-ace6-fad77c35a0b2","Type":"ContainerStarted","Data":"49dcfcca50502f54b8c1c41745057b970b73ef353cd43b4b642a4ae9cce2781a"}
Nov 01 05:15:03.904192 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:03.904120    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7d1ec3-55d75cf7c8-25l2f"
Nov 01 05:15:04.025659 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:04.025540    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7d1ec3-55d75cf7c8-25l2f" podStartSLOduration=2.085834648 podStartE2EDuration="3.025522189s" totalImagesPullingTime="939.687541ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:15:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:15:04.024049992 +0000 UTC m=+682.094846716" watchObservedRunningTime="2026-11-01 05:15:04.025522189 +0000 UTC m=+682.096318910"
Nov 01 05:15:04.582969 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:04.582917    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:15:04.760718 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:04.760684    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:15:05.915845 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:05.915813    5689 generic.go:356] "Generic (PLEG): container finished" podID="02597e35-83cc-42f6-9dea-e1b04942fe9e" containerID="7b89786800ef306d16b8c0388960b236d60f7fd94be328d5ba64f85ef52e92d7" exitCode=60
Nov 01 05:15:05.916268 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:05.915902    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891835-h469d" event={"ID":"02597e35-83cc-42f6-9dea-e1b04942fe9e","Type":"ContainerDied","Data":"7b89786800ef306d16b8c0388960b236d60f7fd94be328d5ba64f85ef52e92d7"}
Nov 01 05:15:05.916268 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:05.916135    5689 scope.go:122] "RemoveContainer" containerID="7b89786800ef306d16b8c0388960b236d60f7fd94be328d5ba64f85ef52e92d7"
Nov 01 05:15:06.179416 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:06.179388    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:15:06.367090 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:06.367052    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:15:06.585696 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:06.585620    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-3d70ff-5ddfbc8c59-7xzdv"]
Nov 01 05:15:06.589201 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:06.589184    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3d70ff-5ddfbc8c59-7xzdv"
Nov 01 05:15:06.643541 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:06.643490    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-3d70ff-serving-cert\""
Nov 01 05:15:06.681483 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:06.681191    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d48b428-8bcb-4777-9e1b-03b8eb99ecfd-metrics-certs\") pod \"maas-api-e2e-worker-w2-3d70ff-5ddfbc8c59-7xzdv\" (UID: \"0d48b428-8bcb-4777-9e1b-03b8eb99ecfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3d70ff-5ddfbc8c59-7xzdv"
Nov 01 05:15:06.681483 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:06.681286    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmgj2\" (UniqueName: \"kubernetes.io/projected/0d48b428-8bcb-4777-9e1b-03b8eb99ecfd-kube-api-access-vmgj2\") pod \"maas-api-e2e-worker-w2-3d70ff-5ddfbc8c59-7xzdv\" (UID: \"0d48b428-8bcb-4777-9e1b-03b8eb99ecfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3d70ff-5ddfbc8c59-7xzdv"
Nov 01 05:15:06.681483 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:06.681319    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d48b428-8bcb-4777-9e1b-03b8eb99ecfd-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-3d70ff-5ddfbc8c59-7xzdv\" (UID: \"0d48b428-8bcb-4777-9e1b-03b8eb99ecfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3d70ff-5ddfbc8c59-7xzdv"
Nov 01 05:15:06.681483 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:06.681363    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d48b428-8bcb-4777-9e1b-03b8eb99ecfd-maas-api-tls\") pod \"maas-api-e2e-worker-w2-3d70ff-5ddfbc8c59-7xzdv\" (UID: \"0d48b428-8bcb-4777-9e1b-03b8eb99ecfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3d70ff-5ddfbc8c59-7xzdv"
Nov 01 05:15:06.708512 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:06.684651    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-3d70ff-5ddfbc8c59-7xzdv"]
Nov 01 05:15:06.782223 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:06.782182    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d48b428-8bcb-4777-9e1b-03b8eb99ecfd-metrics-certs\") pod \"maas-api-e2e-worker-w2-3d70ff-5ddfbc8c59-7xzdv\" (UID: \"0d48b428-8bcb-4777-9e1b-03b8eb99ecfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3d70ff-5ddfbc8c59-7xzdv"
Nov 01 05:15:06.782369 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:06.782277    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmgj2\" (UniqueName: \"kubernetes.io/projected/0d48b428-8bcb-4777-9e1b-03b8eb99ecfd-kube-api-access-vmgj2\") pod \"maas-api-e2e-worker-w2-3d70ff-5ddfbc8c59-7xzdv\" (UID: \"0d48b428-8bcb-4777-9e1b-03b8eb99ecfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3d70ff-5ddfbc8c59-7xzdv"
Nov 01 05:15:06.782369 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:06.782310    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d48b428-8bcb-4777-9e1b-03b8eb99ecfd-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-3d70ff-5ddfbc8c59-7xzdv\" (UID: \"0d48b428-8bcb-4777-9e1b-03b8eb99ecfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3d70ff-5ddfbc8c59-7xzdv"
Nov 01 05:15:06.782707 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:06.782676    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d48b428-8bcb-4777-9e1b-03b8eb99ecfd-maas-api-tls\") pod \"maas-api-e2e-worker-w2-3d70ff-5ddfbc8c59-7xzdv\" (UID: \"0d48b428-8bcb-4777-9e1b-03b8eb99ecfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3d70ff-5ddfbc8c59-7xzdv"
Nov 01 05:15:06.782847 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:15:06.782811    5689 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w2-3d70ff-serving-cert: secret "maas-api-e2e-worker-w2-3d70ff-serving-cert" not found
Nov 01 05:15:06.782922 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:15:06.782881    5689 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d48b428-8bcb-4777-9e1b-03b8eb99ecfd-maas-api-tls podName:0d48b428-8bcb-4777-9e1b-03b8eb99ecfd nodeName:}" failed. No retries permitted until 2026-11-01 05:15:07.28285989 +0000 UTC m=+685.353656607 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0d48b428-8bcb-4777-9e1b-03b8eb99ecfd-maas-api-tls") pod "maas-api-e2e-worker-w2-3d70ff-5ddfbc8c59-7xzdv" (UID: "0d48b428-8bcb-4777-9e1b-03b8eb99ecfd") : secret "maas-api-e2e-worker-w2-3d70ff-serving-cert" not found
Nov 01 05:15:06.783084 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:06.783059    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d48b428-8bcb-4777-9e1b-03b8eb99ecfd-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-3d70ff-5ddfbc8c59-7xzdv\" (UID: \"0d48b428-8bcb-4777-9e1b-03b8eb99ecfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3d70ff-5ddfbc8c59-7xzdv"
Nov 01 05:15:06.785329 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:06.785300    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d48b428-8bcb-4777-9e1b-03b8eb99ecfd-metrics-certs\") pod \"maas-api-e2e-worker-w2-3d70ff-5ddfbc8c59-7xzdv\" (UID: \"0d48b428-8bcb-4777-9e1b-03b8eb99ecfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3d70ff-5ddfbc8c59-7xzdv"
Nov 01 05:15:06.819372 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:06.819343    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:15:06.843615 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:06.843560    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmgj2\" (UniqueName: \"kubernetes.io/projected/0d48b428-8bcb-4777-9e1b-03b8eb99ecfd-kube-api-access-vmgj2\") pod \"maas-api-e2e-worker-w2-3d70ff-5ddfbc8c59-7xzdv\" (UID: \"0d48b428-8bcb-4777-9e1b-03b8eb99ecfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3d70ff-5ddfbc8c59-7xzdv"
Nov 01 05:15:06.923853 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:06.923817    5689 generic.go:356] "Generic (PLEG): container finished" podID="02597e35-83cc-42f6-9dea-e1b04942fe9e" containerID="22bbbe7fdd3e2779b2223e50e1c2df26961452fd6a3e3d95630981da4e99bbec" exitCode=60
Nov 01 05:15:06.924265 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:06.923895    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891835-h469d" event={"ID":"02597e35-83cc-42f6-9dea-e1b04942fe9e","Type":"ContainerDied","Data":"22bbbe7fdd3e2779b2223e50e1c2df26961452fd6a3e3d95630981da4e99bbec"}
Nov 01 05:15:06.924265 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:06.923970    5689 scope.go:122] "RemoveContainer" containerID="7b89786800ef306d16b8c0388960b236d60f7fd94be328d5ba64f85ef52e92d7"
Nov 01 05:15:06.924402 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:06.924290    5689 scope.go:122] "RemoveContainer" containerID="22bbbe7fdd3e2779b2223e50e1c2df26961452fd6a3e3d95630981da4e99bbec"
Nov 01 05:15:06.924605 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:15:06.924587    5689 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891835-h469d_odh-ai-gateway-infra(02597e35-83cc-42f6-9dea-e1b04942fe9e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891835-h469d" podUID="02597e35-83cc-42f6-9dea-e1b04942fe9e"
Nov 01 05:15:07.001408 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:07.001333    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:15:07.294414 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:07.294381    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d48b428-8bcb-4777-9e1b-03b8eb99ecfd-maas-api-tls\") pod \"maas-api-e2e-worker-w2-3d70ff-5ddfbc8c59-7xzdv\" (UID: \"0d48b428-8bcb-4777-9e1b-03b8eb99ecfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3d70ff-5ddfbc8c59-7xzdv"
Nov 01 05:15:07.297040 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:07.297021    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d48b428-8bcb-4777-9e1b-03b8eb99ecfd-maas-api-tls\") pod \"maas-api-e2e-worker-w2-3d70ff-5ddfbc8c59-7xzdv\" (UID: \"0d48b428-8bcb-4777-9e1b-03b8eb99ecfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3d70ff-5ddfbc8c59-7xzdv"
Nov 01 05:15:07.502286 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:07.502247    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3d70ff-5ddfbc8c59-7xzdv"
Nov 01 05:15:07.876983 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:07.876896    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-3d70ff-5ddfbc8c59-7xzdv"]
Nov 01 05:15:07.879756 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:15:07.879729    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d48b428_8bcb_4777_9e1b_03b8eb99ecfd.slice/crio-5d4ee91c66286af344b406038418f8176d15905ab0e13f3f905748577937a7d7 WatchSource:0}: Error finding container 5d4ee91c66286af344b406038418f8176d15905ab0e13f3f905748577937a7d7: Status 404 returned error can't find the container with id 5d4ee91c66286af344b406038418f8176d15905ab0e13f3f905748577937a7d7
Nov 01 05:15:07.930056 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:07.930031    5689 scope.go:122] "RemoveContainer" containerID="22bbbe7fdd3e2779b2223e50e1c2df26961452fd6a3e3d95630981da4e99bbec"
Nov 01 05:15:07.930485 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:15:07.930318    5689 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891835-h469d_odh-ai-gateway-infra(02597e35-83cc-42f6-9dea-e1b04942fe9e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891835-h469d" podUID="02597e35-83cc-42f6-9dea-e1b04942fe9e"
Nov 01 05:15:07.931067 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:07.931042    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3d70ff-5ddfbc8c59-7xzdv" event={"ID":"0d48b428-8bcb-4777-9e1b-03b8eb99ecfd","Type":"ContainerStarted","Data":"5d4ee91c66286af344b406038418f8176d15905ab0e13f3f905748577937a7d7"}
Nov 01 05:15:08.937572 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:08.937431    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3d70ff-5ddfbc8c59-7xzdv" event={"ID":"0d48b428-8bcb-4777-9e1b-03b8eb99ecfd","Type":"ContainerStarted","Data":"9faac4731985ad32caf58e7bb4d2f7bf20ee8215d057ce80c9f0bbf557cd0819"}
Nov 01 05:15:08.937931 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:08.937639    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3d70ff-5ddfbc8c59-7xzdv"
Nov 01 05:15:09.056036 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:09.055944    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3d70ff-5ddfbc8c59-7xzdv" podStartSLOduration=2.186764009 podStartE2EDuration="3.055930846s" totalImagesPullingTime="869.166837ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:15:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:15:09.054213421 +0000 UTC m=+687.125010178" watchObservedRunningTime="2026-11-01 05:15:09.055930846 +0000 UTC m=+687.126727565"
Nov 01 05:15:09.915224 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:09.915196    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7d1ec3-55d75cf7c8-25l2f"
Nov 01 05:15:12.067219 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:12.067187    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:15:12.249834 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:12.249791    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:15:12.430951 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:12.430913    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:15:12.804485 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:12.804379    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:15:14.947588 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:14.947553    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3d70ff-5ddfbc8c59-7xzdv"
Nov 01 05:15:17.018478 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:17.016856    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:15:20.060901 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:20.060866    5689 scope.go:122] "RemoveContainer" containerID="22bbbe7fdd3e2779b2223e50e1c2df26961452fd6a3e3d95630981da4e99bbec"
Nov 01 05:15:21.038518 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:21.038442    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891835-h469d" event={"ID":"02597e35-83cc-42f6-9dea-e1b04942fe9e","Type":"ContainerStarted","Data":"1a46afd884021cba41f442efa4d9c5c0e31fe4a9863a0dbd1cb48f824f76f5b1"}
Nov 01 05:15:21.158637 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:21.158557    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891835-h469d" podStartSLOduration=16.985176782 podStartE2EDuration="21.158536728s" totalImagesPullingTime="4.173359946s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:15:21.156439662 +0000 UTC m=+699.227236382" watchObservedRunningTime="2026-11-01 05:15:21.158536728 +0000 UTC m=+699.229333449"
Nov 01 05:15:21.923592 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:21.923526    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:15:22.122478 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:22.111679    5689 generic.go:356] "Generic (PLEG): container finished" podID="02597e35-83cc-42f6-9dea-e1b04942fe9e" containerID="1a46afd884021cba41f442efa4d9c5c0e31fe4a9863a0dbd1cb48f824f76f5b1" exitCode=60
Nov 01 05:15:22.122478 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:22.111833    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891835-h469d" event={"ID":"02597e35-83cc-42f6-9dea-e1b04942fe9e","Type":"ContainerDied","Data":"1a46afd884021cba41f442efa4d9c5c0e31fe4a9863a0dbd1cb48f824f76f5b1"}
Nov 01 05:15:22.122478 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:22.111888    5689 scope.go:122] "RemoveContainer" containerID="22bbbe7fdd3e2779b2223e50e1c2df26961452fd6a3e3d95630981da4e99bbec"
Nov 01 05:15:22.122478 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:22.112307    5689 scope.go:122] "RemoveContainer" containerID="1a46afd884021cba41f442efa4d9c5c0e31fe4a9863a0dbd1cb48f824f76f5b1"
Nov 01 05:15:22.122478 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:15:22.112571    5689 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29891835-h469d_odh-ai-gateway-infra(02597e35-83cc-42f6-9dea-e1b04942fe9e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891835-h469d" podUID="02597e35-83cc-42f6-9dea-e1b04942fe9e"
Nov 01 05:15:22.169327 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:22.169234    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891835-h469d"]
Nov 01 05:15:23.995989 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:23.995672    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891835-h469d"
Nov 01 05:15:24.057076 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:24.057040    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02597e35-83cc-42f6-9dea-e1b04942fe9e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02597e35-83cc-42f6-9dea-e1b04942fe9e-maas-api-service-ca\") pod \"02597e35-83cc-42f6-9dea-e1b04942fe9e\" (UID: \"02597e35-83cc-42f6-9dea-e1b04942fe9e\") "
Nov 01 05:15:24.057247 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:24.057217    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02597e35-83cc-42f6-9dea-e1b04942fe9e-kube-api-access-24kbf\" (UniqueName: \"kubernetes.io/projected/02597e35-83cc-42f6-9dea-e1b04942fe9e-kube-api-access-24kbf\") pod \"02597e35-83cc-42f6-9dea-e1b04942fe9e\" (UID: \"02597e35-83cc-42f6-9dea-e1b04942fe9e\") "
Nov 01 05:15:24.057420 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:24.057396    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02597e35-83cc-42f6-9dea-e1b04942fe9e-maas-api-service-ca" pod "02597e35-83cc-42f6-9dea-e1b04942fe9e" (UID: "02597e35-83cc-42f6-9dea-e1b04942fe9e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:15:24.057647 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:24.057629    5689 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02597e35-83cc-42f6-9dea-e1b04942fe9e-maas-api-service-ca\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:15:24.059415 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:24.059393    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02597e35-83cc-42f6-9dea-e1b04942fe9e-kube-api-access-24kbf" pod "02597e35-83cc-42f6-9dea-e1b04942fe9e" (UID: "02597e35-83cc-42f6-9dea-e1b04942fe9e"). InnerVolumeSpecName "kube-api-access-24kbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:15:24.151223 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:24.151181    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891835-h469d" event={"ID":"02597e35-83cc-42f6-9dea-e1b04942fe9e","Type":"ContainerDied","Data":"bda6b01cdb5664ee7f62f95ae5d75d42be96be5acdebcb88051f32d204f714e3"}
Nov 01 05:15:24.151494 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:24.151255    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891835-h469d"
Nov 01 05:15:24.151641 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:24.151622    5689 scope.go:122] "RemoveContainer" containerID="1a46afd884021cba41f442efa4d9c5c0e31fe4a9863a0dbd1cb48f824f76f5b1"
Nov 01 05:15:24.159129 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:24.159103    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24kbf\" (UniqueName: \"kubernetes.io/projected/02597e35-83cc-42f6-9dea-e1b04942fe9e-kube-api-access-24kbf\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:15:24.315582 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:24.315507    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891835-h469d"]
Nov 01 05:15:24.319098 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:24.319068    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891835-h469d"]
Nov 01 05:15:24.764098 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:24.764068    5689 patch_prober.go:29] interesting pod/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": read tcp 10.134.0.2:34502->10.134.0.43:15021: read: connection reset by peer" start-of-body=
Nov 01 05:15:24.764365 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:24.764141    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg" podUID="ba32daa8-34ac-44b9-a090-d4804cdeed41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": read tcp 10.134.0.2:34502->10.134.0.43:15021: read: connection reset by peer"
Nov 01 05:15:24.851937 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:24.851901    5689 patch_prober.go:29] interesting pod/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Nov 01 05:15:24.852057 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:24.851951    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb" podUID="c8a582fb-747c-4c55-a0e3-b5f5139fa28f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Nov 01 05:15:25.158185 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:25.158136    5689 generic.go:356] "Generic (PLEG): container finished" podID="c8a582fb-747c-4c55-a0e3-b5f5139fa28f" containerID="7cadc1665174645d53c05fbe46ab5b2ab55f637c9e8ac49e4e497655859406a2" exitCode=0
Nov 01 05:15:25.158676 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:25.158297    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb" event={"ID":"c8a582fb-747c-4c55-a0e3-b5f5139fa28f","Type":"ContainerDied","Data":"7cadc1665174645d53c05fbe46ab5b2ab55f637c9e8ac49e4e497655859406a2"}
Nov 01 05:15:25.158733 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:25.158694    5689 scope.go:122] "RemoveContainer" containerID="7cadc1665174645d53c05fbe46ab5b2ab55f637c9e8ac49e4e497655859406a2"
Nov 01 05:15:25.159969 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:25.159936    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:15:25.160201 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:25.160162    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:15:25.160305 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:25.160224    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:15:25.160389 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:25.160367    5689 generic.go:356] "Generic (PLEG): container finished" podID="ba32daa8-34ac-44b9-a090-d4804cdeed41" containerID="e20fe1cec4efa6ca8c184e93cc6b2de1917576e46459be9b2c80b3b9130645c5" exitCode=0
Nov 01 05:15:25.160529 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:25.160470    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg" event={"ID":"ba32daa8-34ac-44b9-a090-d4804cdeed41","Type":"ContainerDied","Data":"e20fe1cec4efa6ca8c184e93cc6b2de1917576e46459be9b2c80b3b9130645c5"}
Nov 01 05:15:25.160854 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:25.160836    5689 scope.go:122] "RemoveContainer" containerID="e20fe1cec4efa6ca8c184e93cc6b2de1917576e46459be9b2c80b3b9130645c5"
Nov 01 05:15:25.161916 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:25.161886    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:15:25.162004 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:25.161957    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:15:25.162004 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:25.161989    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:15:26.074636 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:26.074089    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02597e35-83cc-42f6-9dea-e1b04942fe9e" path="/var/lib/kubelet/pods/02597e35-83cc-42f6-9dea-e1b04942fe9e/volumes"
Nov 01 05:15:26.168606 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:26.168572    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb" event={"ID":"c8a582fb-747c-4c55-a0e3-b5f5139fa28f","Type":"ContainerStarted","Data":"977315bedf667832fb45730d9da5f32d29e00bd9ffcc867e2b1fef2900eb0767"}
Nov 01 05:15:26.170942 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:26.170915    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg" event={"ID":"ba32daa8-34ac-44b9-a090-d4804cdeed41","Type":"ContainerStarted","Data":"4505d6a083ef4eea3dadeb0a38fdafc33b63545dd3bc826114ca10b7d9af344d"}
Nov 01 05:15:26.198477 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:26.197886    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg"
Nov 01 05:15:26.198477 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:26.198020    5689 patch_prober.go:29] interesting pod/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Nov 01 05:15:26.198477 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:26.198065    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg" podUID="ba32daa8-34ac-44b9-a090-d4804cdeed41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Nov 01 05:15:26.809483 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:26.809295    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:15:26.865169 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:26.865128    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb"
Nov 01 05:15:26.865361 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:26.865247    5689 patch_prober.go:29] interesting pod/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Nov 01 05:15:26.865361 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:26.865284    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb" podUID="c8a582fb-747c-4c55-a0e3-b5f5139fa28f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Nov 01 05:15:27.195238 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:27.194813    5689 patch_prober.go:29] interesting pod/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Nov 01 05:15:27.195238 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:27.194870    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg" podUID="ba32daa8-34ac-44b9-a090-d4804cdeed41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Nov 01 05:15:27.864728 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:27.864696    5689 patch_prober.go:29] interesting pod/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Nov 01 05:15:27.864903 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:27.864748    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb" podUID="c8a582fb-747c-4c55-a0e3-b5f5139fa28f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Nov 01 05:15:28.195489 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:28.194858    5689 patch_prober.go:29] interesting pod/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Nov 01 05:15:28.195489 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:28.194924    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg" podUID="ba32daa8-34ac-44b9-a090-d4804cdeed41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Nov 01 05:15:28.226851 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:28.226808    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:15:28.865145 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:28.865113    5689 patch_prober.go:29] interesting pod/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Nov 01 05:15:28.865318 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:28.865159    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb" podUID="c8a582fb-747c-4c55-a0e3-b5f5139fa28f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Nov 01 05:15:29.203502 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:29.203441    5689 patch_prober.go:29] interesting pod/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Nov 01 05:15:29.204095 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:29.204065    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg" podUID="ba32daa8-34ac-44b9-a090-d4804cdeed41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Nov 01 05:15:29.865609 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:29.865570    5689 patch_prober.go:29] interesting pod/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Nov 01 05:15:29.865781 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:29.865622    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb" podUID="c8a582fb-747c-4c55-a0e3-b5f5139fa28f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Nov 01 05:15:30.069178 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:15:30.069146    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:15:30.196475 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:30.195833    5689 patch_prober.go:29] interesting pod/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Nov 01 05:15:30.196475 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:30.195896    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg" podUID="ba32daa8-34ac-44b9-a090-d4804cdeed41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Nov 01 05:15:30.864951 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:30.864909    5689 patch_prober.go:29] interesting pod/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Nov 01 05:15:30.865347 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:30.864972    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb" podUID="c8a582fb-747c-4c55-a0e3-b5f5139fa28f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Nov 01 05:15:31.025400 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:31.025358    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:15:31.195115 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:31.195071    5689 patch_prober.go:29] interesting pod/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Nov 01 05:15:31.195286 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:31.195159    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg" podUID="ba32daa8-34ac-44b9-a090-d4804cdeed41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Nov 01 05:15:31.697553 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:31.697503    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:15:31.872991 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:31.872946    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb"
Nov 01 05:15:31.874367 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:31.874342    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb"
Nov 01 05:15:31.875315 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:31.875292    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb"
Nov 01 05:15:31.877045 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:31.877023    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:15:32.199125 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:32.199089    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg"
Nov 01 05:15:32.199605 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:32.199579    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg"
Nov 01 05:15:32.200207 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:32.200186    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg"
Nov 01 05:15:35.001050 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:35.000990    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:15:36.316383 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:36.316351    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:15:36.827488 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:36.827430    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:15:37.009033 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:37.008995    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:15:37.185959 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:37.185917    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:15:39.099170 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:39.099140    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:15:40.733690 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:40.733644    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:15:41.379805 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:41.379773    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:15:41.573685 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:41.573644    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:15:42.012008 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:42.011976    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:15:46.805141 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:46.805106    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:15:48.051261 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:48.051231    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:15:48.670285 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:48.670249    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:15:48.882079 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:48.882046    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:15:51.695313 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:51.695233    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:15:52.548277 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:52.548244    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:15:52.732607 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:52.732577    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:15:52.913167 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:52.913141    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:15:55.786700 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:55.786666    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:15:55.967711 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:55.967678    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:15:56.326249 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:56.326216    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2ea125-5cc577d6b5-p45rd"]
Nov 01 05:15:56.326912 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:56.326888    5689 cpu_manager.go:414] "containerMap: removing container" podUID="02597e35-83cc-42f6-9dea-e1b04942fe9e" containerName="cleanup"
Nov 01 05:15:56.327028 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:56.326914    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02597e35-83cc-42f6-9dea-e1b04942fe9e" containerName="cleanup"
Nov 01 05:15:56.327028 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:56.326946    5689 cpu_manager.go:414] "containerMap: removing container" podUID="02597e35-83cc-42f6-9dea-e1b04942fe9e" containerName="cleanup"
Nov 01 05:15:56.327028 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:56.326954    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02597e35-83cc-42f6-9dea-e1b04942fe9e" containerName="cleanup"
Nov 01 05:15:56.327028 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:56.326974    5689 cpu_manager.go:414] "containerMap: removing container" podUID="02597e35-83cc-42f6-9dea-e1b04942fe9e" containerName="cleanup"
Nov 01 05:15:56.327028 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:56.326982    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02597e35-83cc-42f6-9dea-e1b04942fe9e" containerName="cleanup"
Nov 01 05:15:56.327199 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:56.327072    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="02597e35-83cc-42f6-9dea-e1b04942fe9e" containerName="cleanup"
Nov 01 05:15:56.327199 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:56.327085    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="02597e35-83cc-42f6-9dea-e1b04942fe9e" containerName="cleanup"
Nov 01 05:15:56.331515 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:56.331490    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2ea125-5cc577d6b5-p45rd"
Nov 01 05:15:56.384124 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:56.384089    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2ea125-5cc577d6b5-p45rd"]
Nov 01 05:15:56.430354 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:56.430317    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4wbm\" (UniqueName: \"kubernetes.io/projected/731cd5ef-5a2a-4771-8067-75b133315fbf-kube-api-access-j4wbm\") pod \"payload-processing-e2e-inf-a-2ea125-5cc577d6b5-p45rd\" (UID: \"731cd5ef-5a2a-4771-8067-75b133315fbf\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2ea125-5cc577d6b5-p45rd"
Nov 01 05:15:56.430529 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:56.430438    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/731cd5ef-5a2a-4771-8067-75b133315fbf-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2ea125-5cc577d6b5-p45rd\" (UID: \"731cd5ef-5a2a-4771-8067-75b133315fbf\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2ea125-5cc577d6b5-p45rd"
Nov 01 05:15:56.431723 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:56.431701    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-2ea125\""
Nov 01 05:15:56.431827 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:56.431784    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-2ea125-dockercfg-d85wp\""
Nov 01 05:15:56.531616 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:56.531589    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4wbm\" (UniqueName: \"kubernetes.io/projected/731cd5ef-5a2a-4771-8067-75b133315fbf-kube-api-access-j4wbm\") pod \"payload-processing-e2e-inf-a-2ea125-5cc577d6b5-p45rd\" (UID: \"731cd5ef-5a2a-4771-8067-75b133315fbf\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2ea125-5cc577d6b5-p45rd"
Nov 01 05:15:56.531787 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:56.531658    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/731cd5ef-5a2a-4771-8067-75b133315fbf-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2ea125-5cc577d6b5-p45rd\" (UID: \"731cd5ef-5a2a-4771-8067-75b133315fbf\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2ea125-5cc577d6b5-p45rd"
Nov 01 05:15:56.532372 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:56.532354    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/731cd5ef-5a2a-4771-8067-75b133315fbf-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2ea125-5cc577d6b5-p45rd\" (UID: \"731cd5ef-5a2a-4771-8067-75b133315fbf\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2ea125-5cc577d6b5-p45rd"
Nov 01 05:15:56.592393 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:56.592335    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4wbm\" (UniqueName: \"kubernetes.io/projected/731cd5ef-5a2a-4771-8067-75b133315fbf-kube-api-access-j4wbm\") pod \"payload-processing-e2e-inf-a-2ea125-5cc577d6b5-p45rd\" (UID: \"731cd5ef-5a2a-4771-8067-75b133315fbf\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2ea125-5cc577d6b5-p45rd"
Nov 01 05:15:56.655947 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:56.655915    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2ea125-5cc577d6b5-p45rd"
Nov 01 05:15:57.017746 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:57.017715    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:15:57.037595 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:57.033530    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2ea125-5cc577d6b5-p45rd"]
Nov 01 05:15:57.041846 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:15:57.041801    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod731cd5ef_5a2a_4771_8067_75b133315fbf.slice/crio-67bc233fef281420d2ba8975aa358b31f536c8281de8b124823e96f32d19c533 WatchSource:0}: Error finding container 67bc233fef281420d2ba8975aa358b31f536c8281de8b124823e96f32d19c533: Status 404 returned error can't find the container with id 67bc233fef281420d2ba8975aa358b31f536c8281de8b124823e96f32d19c533
Nov 01 05:15:57.372031 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:57.371952    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2ea125-5cc577d6b5-p45rd" event={"ID":"731cd5ef-5a2a-4771-8067-75b133315fbf","Type":"ContainerStarted","Data":"67bc233fef281420d2ba8975aa358b31f536c8281de8b124823e96f32d19c533"}
Nov 01 05:15:57.543230 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:15:57.543196    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:16:00.068920 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:16:00.068829    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:16:00.388593 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:00.388496    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2ea125-5cc577d6b5-p45rd" event={"ID":"731cd5ef-5a2a-4771-8067-75b133315fbf","Type":"ContainerStarted","Data":"38148b014b4d51fd0330081708bf86333cd0f87d4b882076f7fd320c3eec3c57"}
Nov 01 05:16:00.388593 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:00.388580    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-2ea125-5cc577d6b5-p45rd"
Nov 01 05:16:00.505179 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:00.505129    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-2ea125-5cc577d6b5-p45rd" podStartSLOduration=1.842469684 podStartE2EDuration="4.50511738s" totalImagesPullingTime="2.662647696s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:15:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:16:00.503445063 +0000 UTC m=+738.574241774" watchObservedRunningTime="2026-11-01 05:16:00.50511738 +0000 UTC m=+738.575914129"
Nov 01 05:16:00.754525 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:00.754492    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-3d70ff/simulator-w2-3d70ff-kserve-898c5fb4f-t5bzk"]
Nov 01 05:16:00.754994 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:00.754982    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="02597e35-83cc-42f6-9dea-e1b04942fe9e" containerName="cleanup"
Nov 01 05:16:00.758206 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:00.758191    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-3d70ff/simulator-w2-3d70ff-kserve-898c5fb4f-t5bzk"
Nov 01 05:16:00.810278 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:00.810254    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-3d70ff\"/\"default-dockercfg-rz4dl\""
Nov 01 05:16:00.810278 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:00.810279    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-3d70ff\"/\"simulator-w2-3d70ff-kserve-self-signed-certs\""
Nov 01 05:16:00.810495 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:00.810279    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-3d70ff\"/\"kube-root-ca.crt\""
Nov 01 05:16:00.810495 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:00.810340    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-3d70ff\"/\"openshift-service-ca.crt\""
Nov 01 05:16:00.817341 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:00.817313    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-3d70ff/simulator-w2-3d70ff-kserve-898c5fb4f-t5bzk"]
Nov 01 05:16:00.874104 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:00.874073    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:16:00.879333 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:00.879313    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca-model-cache\") pod \"simulator-w2-3d70ff-kserve-898c5fb4f-t5bzk\" (UID: \"05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca\") " pod="e2e-models-e2e-worker-w2-3d70ff/simulator-w2-3d70ff-kserve-898c5fb4f-t5bzk"
Nov 01 05:16:00.879433 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:00.879375    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca-tmp-dir\") pod \"simulator-w2-3d70ff-kserve-898c5fb4f-t5bzk\" (UID: \"05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca\") " pod="e2e-models-e2e-worker-w2-3d70ff/simulator-w2-3d70ff-kserve-898c5fb4f-t5bzk"
Nov 01 05:16:00.879556 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:00.879539    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca-tls-certs\") pod \"simulator-w2-3d70ff-kserve-898c5fb4f-t5bzk\" (UID: \"05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca\") " pod="e2e-models-e2e-worker-w2-3d70ff/simulator-w2-3d70ff-kserve-898c5fb4f-t5bzk"
Nov 01 05:16:00.879652 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:00.879637    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca-home\") pod \"simulator-w2-3d70ff-kserve-898c5fb4f-t5bzk\" (UID: \"05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca\") " pod="e2e-models-e2e-worker-w2-3d70ff/simulator-w2-3d70ff-kserve-898c5fb4f-t5bzk"
Nov 01 05:16:00.879715 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:00.879677    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqkd6\" (UniqueName: \"kubernetes.io/projected/05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca-kube-api-access-hqkd6\") pod \"simulator-w2-3d70ff-kserve-898c5fb4f-t5bzk\" (UID: \"05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca\") " pod="e2e-models-e2e-worker-w2-3d70ff/simulator-w2-3d70ff-kserve-898c5fb4f-t5bzk"
Nov 01 05:16:00.879773 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:00.879762    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca-dshm\") pod \"simulator-w2-3d70ff-kserve-898c5fb4f-t5bzk\" (UID: \"05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca\") " pod="e2e-models-e2e-worker-w2-3d70ff/simulator-w2-3d70ff-kserve-898c5fb4f-t5bzk"
Nov 01 05:16:00.980532 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:00.980499    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca-home\") pod \"simulator-w2-3d70ff-kserve-898c5fb4f-t5bzk\" (UID: \"05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca\") " pod="e2e-models-e2e-worker-w2-3d70ff/simulator-w2-3d70ff-kserve-898c5fb4f-t5bzk"
Nov 01 05:16:00.980702 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:00.980541    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqkd6\" (UniqueName: \"kubernetes.io/projected/05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca-kube-api-access-hqkd6\") pod \"simulator-w2-3d70ff-kserve-898c5fb4f-t5bzk\" (UID: \"05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca\") " pod="e2e-models-e2e-worker-w2-3d70ff/simulator-w2-3d70ff-kserve-898c5fb4f-t5bzk"
Nov 01 05:16:00.980702 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:00.980577    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca-dshm\") pod \"simulator-w2-3d70ff-kserve-898c5fb4f-t5bzk\" (UID: \"05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca\") " pod="e2e-models-e2e-worker-w2-3d70ff/simulator-w2-3d70ff-kserve-898c5fb4f-t5bzk"
Nov 01 05:16:00.980796 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:00.980704    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca-model-cache\") pod \"simulator-w2-3d70ff-kserve-898c5fb4f-t5bzk\" (UID: \"05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca\") " pod="e2e-models-e2e-worker-w2-3d70ff/simulator-w2-3d70ff-kserve-898c5fb4f-t5bzk"
Nov 01 05:16:00.980851 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:00.980800    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca-tmp-dir\") pod \"simulator-w2-3d70ff-kserve-898c5fb4f-t5bzk\" (UID: \"05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca\") " pod="e2e-models-e2e-worker-w2-3d70ff/simulator-w2-3d70ff-kserve-898c5fb4f-t5bzk"
Nov 01 05:16:00.980900 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:00.980861    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca-tls-certs\") pod \"simulator-w2-3d70ff-kserve-898c5fb4f-t5bzk\" (UID: \"05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca\") " pod="e2e-models-e2e-worker-w2-3d70ff/simulator-w2-3d70ff-kserve-898c5fb4f-t5bzk"
Nov 01 05:16:00.980958 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:00.980936    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca-home\") pod \"simulator-w2-3d70ff-kserve-898c5fb4f-t5bzk\" (UID: \"05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca\") " pod="e2e-models-e2e-worker-w2-3d70ff/simulator-w2-3d70ff-kserve-898c5fb4f-t5bzk"
Nov 01 05:16:00.981071 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:00.981049    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca-model-cache\") pod \"simulator-w2-3d70ff-kserve-898c5fb4f-t5bzk\" (UID: \"05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca\") " pod="e2e-models-e2e-worker-w2-3d70ff/simulator-w2-3d70ff-kserve-898c5fb4f-t5bzk"
Nov 01 05:16:00.981136 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:00.981091    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca-tmp-dir\") pod \"simulator-w2-3d70ff-kserve-898c5fb4f-t5bzk\" (UID: \"05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca\") " pod="e2e-models-e2e-worker-w2-3d70ff/simulator-w2-3d70ff-kserve-898c5fb4f-t5bzk"
Nov 01 05:16:00.982958 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:00.982933    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca-dshm\") pod \"simulator-w2-3d70ff-kserve-898c5fb4f-t5bzk\" (UID: \"05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca\") " pod="e2e-models-e2e-worker-w2-3d70ff/simulator-w2-3d70ff-kserve-898c5fb4f-t5bzk"
Nov 01 05:16:00.983499 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:00.983477    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca-tls-certs\") pod \"simulator-w2-3d70ff-kserve-898c5fb4f-t5bzk\" (UID: \"05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca\") " pod="e2e-models-e2e-worker-w2-3d70ff/simulator-w2-3d70ff-kserve-898c5fb4f-t5bzk"
Nov 01 05:16:01.040919 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:01.040858    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqkd6\" (UniqueName: \"kubernetes.io/projected/05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca-kube-api-access-hqkd6\") pod \"simulator-w2-3d70ff-kserve-898c5fb4f-t5bzk\" (UID: \"05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca\") " pod="e2e-models-e2e-worker-w2-3d70ff/simulator-w2-3d70ff-kserve-898c5fb4f-t5bzk"
Nov 01 05:16:01.081992 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:01.081962    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-3d70ff/simulator-w2-3d70ff-kserve-898c5fb4f-t5bzk"
Nov 01 05:16:01.455975 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:01.455943    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-3d70ff/simulator-w2-3d70ff-kserve-898c5fb4f-t5bzk"]
Nov 01 05:16:01.458866 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:16:01.458839    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod05b70f5f_3a0f_439b_9f0b_a1e9e76ab4ca.slice/crio-67458bc19b71123ee29836a96acf684108e2f6f9a442250d2612018d1a9f6fc8 WatchSource:0}: Error finding container 67458bc19b71123ee29836a96acf684108e2f6f9a442250d2612018d1a9f6fc8: Status 404 returned error can't find the container with id 67458bc19b71123ee29836a96acf684108e2f6f9a442250d2612018d1a9f6fc8
Nov 01 05:16:01.822527 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:01.815878    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:16:02.398492 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:02.398441    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-3d70ff/simulator-w2-3d70ff-kserve-898c5fb4f-t5bzk" event={"ID":"05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca","Type":"ContainerStarted","Data":"eb663254760978ac1c5ae561a4582a7bfe3186f5a4e0e410679c39fb34422a64"}
Nov 01 05:16:02.398879 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:02.398503    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-3d70ff/simulator-w2-3d70ff-kserve-898c5fb4f-t5bzk" event={"ID":"05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca","Type":"ContainerStarted","Data":"67458bc19b71123ee29836a96acf684108e2f6f9a442250d2612018d1a9f6fc8"}
Nov 01 05:16:02.515281 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:02.515228    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-3d70ff/simulator-w2-3d70ff-kserve-898c5fb4f-t5bzk" podStartSLOduration=2.515214223 podStartE2EDuration="2.515214223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:16:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:16:02.513443452 +0000 UTC m=+740.584240172" watchObservedRunningTime="2026-11-01 05:16:02.515214223 +0000 UTC m=+740.586010943"
Nov 01 05:16:03.083027 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:03.082997    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-3d70ff/simulator-w2-3d70ff-kserve-898c5fb4f-t5bzk"
Nov 01 05:16:04.680072 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:04.680039    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-a48099/simulator-w4-a48099-kserve-7df86f7d-rdlrm"]
Nov 01 05:16:04.683686 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:04.683665    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-a48099/simulator-w4-a48099-kserve-7df86f7d-rdlrm"
Nov 01 05:16:04.719596 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:04.719565    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12c3ad65-a804-429f-ab42-9cc7b2433855-dshm\") pod \"simulator-w4-a48099-kserve-7df86f7d-rdlrm\" (UID: \"12c3ad65-a804-429f-ab42-9cc7b2433855\") " pod="e2e-models-e2e-worker-w4-a48099/simulator-w4-a48099-kserve-7df86f7d-rdlrm"
Nov 01 05:16:04.719768 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:04.719613    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12c3ad65-a804-429f-ab42-9cc7b2433855-tmp-dir\") pod \"simulator-w4-a48099-kserve-7df86f7d-rdlrm\" (UID: \"12c3ad65-a804-429f-ab42-9cc7b2433855\") " pod="e2e-models-e2e-worker-w4-a48099/simulator-w4-a48099-kserve-7df86f7d-rdlrm"
Nov 01 05:16:04.719768 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:04.719720    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12c3ad65-a804-429f-ab42-9cc7b2433855-home\") pod \"simulator-w4-a48099-kserve-7df86f7d-rdlrm\" (UID: \"12c3ad65-a804-429f-ab42-9cc7b2433855\") " pod="e2e-models-e2e-worker-w4-a48099/simulator-w4-a48099-kserve-7df86f7d-rdlrm"
Nov 01 05:16:04.719921 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:04.719852    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12c3ad65-a804-429f-ab42-9cc7b2433855-tls-certs\") pod \"simulator-w4-a48099-kserve-7df86f7d-rdlrm\" (UID: \"12c3ad65-a804-429f-ab42-9cc7b2433855\") " pod="e2e-models-e2e-worker-w4-a48099/simulator-w4-a48099-kserve-7df86f7d-rdlrm"
Nov 01 05:16:04.719921 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:04.719884    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12c3ad65-a804-429f-ab42-9cc7b2433855-model-cache\") pod \"simulator-w4-a48099-kserve-7df86f7d-rdlrm\" (UID: \"12c3ad65-a804-429f-ab42-9cc7b2433855\") " pod="e2e-models-e2e-worker-w4-a48099/simulator-w4-a48099-kserve-7df86f7d-rdlrm"
Nov 01 05:16:04.719921 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:04.719913    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kstcx\" (UniqueName: \"kubernetes.io/projected/12c3ad65-a804-429f-ab42-9cc7b2433855-kube-api-access-kstcx\") pod \"simulator-w4-a48099-kserve-7df86f7d-rdlrm\" (UID: \"12c3ad65-a804-429f-ab42-9cc7b2433855\") " pod="e2e-models-e2e-worker-w4-a48099/simulator-w4-a48099-kserve-7df86f7d-rdlrm"
Nov 01 05:16:04.785554 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:04.785527    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-a48099\"/\"simulator-w4-a48099-kserve-self-signed-certs\""
Nov 01 05:16:04.785730 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:04.785637    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-a48099\"/\"default-dockercfg-r7p77\""
Nov 01 05:16:04.785730 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:04.785691    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-a48099\"/\"openshift-service-ca.crt\""
Nov 01 05:16:04.785730 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:04.785694    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-a48099\"/\"kube-root-ca.crt\""
Nov 01 05:16:04.787346 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:04.787328    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-a48099/simulator-w4-a48099-kserve-7df86f7d-rdlrm"]
Nov 01 05:16:04.815797 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:04.815759    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:16:04.821294 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:04.821266    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12c3ad65-a804-429f-ab42-9cc7b2433855-tls-certs\") pod \"simulator-w4-a48099-kserve-7df86f7d-rdlrm\" (UID: \"12c3ad65-a804-429f-ab42-9cc7b2433855\") " pod="e2e-models-e2e-worker-w4-a48099/simulator-w4-a48099-kserve-7df86f7d-rdlrm"
Nov 01 05:16:04.821419 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:04.821306    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12c3ad65-a804-429f-ab42-9cc7b2433855-model-cache\") pod \"simulator-w4-a48099-kserve-7df86f7d-rdlrm\" (UID: \"12c3ad65-a804-429f-ab42-9cc7b2433855\") " pod="e2e-models-e2e-worker-w4-a48099/simulator-w4-a48099-kserve-7df86f7d-rdlrm"
Nov 01 05:16:04.821419 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:04.821344    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kstcx\" (UniqueName: \"kubernetes.io/projected/12c3ad65-a804-429f-ab42-9cc7b2433855-kube-api-access-kstcx\") pod \"simulator-w4-a48099-kserve-7df86f7d-rdlrm\" (UID: \"12c3ad65-a804-429f-ab42-9cc7b2433855\") " pod="e2e-models-e2e-worker-w4-a48099/simulator-w4-a48099-kserve-7df86f7d-rdlrm"
Nov 01 05:16:04.821561 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:04.821469    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12c3ad65-a804-429f-ab42-9cc7b2433855-dshm\") pod \"simulator-w4-a48099-kserve-7df86f7d-rdlrm\" (UID: \"12c3ad65-a804-429f-ab42-9cc7b2433855\") " pod="e2e-models-e2e-worker-w4-a48099/simulator-w4-a48099-kserve-7df86f7d-rdlrm"
Nov 01 05:16:04.821561 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:04.821515    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12c3ad65-a804-429f-ab42-9cc7b2433855-tmp-dir\") pod \"simulator-w4-a48099-kserve-7df86f7d-rdlrm\" (UID: \"12c3ad65-a804-429f-ab42-9cc7b2433855\") " pod="e2e-models-e2e-worker-w4-a48099/simulator-w4-a48099-kserve-7df86f7d-rdlrm"
Nov 01 05:16:04.821561 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:04.821544    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12c3ad65-a804-429f-ab42-9cc7b2433855-home\") pod \"simulator-w4-a48099-kserve-7df86f7d-rdlrm\" (UID: \"12c3ad65-a804-429f-ab42-9cc7b2433855\") " pod="e2e-models-e2e-worker-w4-a48099/simulator-w4-a48099-kserve-7df86f7d-rdlrm"
Nov 01 05:16:04.821976 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:04.821953    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12c3ad65-a804-429f-ab42-9cc7b2433855-home\") pod \"simulator-w4-a48099-kserve-7df86f7d-rdlrm\" (UID: \"12c3ad65-a804-429f-ab42-9cc7b2433855\") " pod="e2e-models-e2e-worker-w4-a48099/simulator-w4-a48099-kserve-7df86f7d-rdlrm"
Nov 01 05:16:04.822527 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:04.822505    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12c3ad65-a804-429f-ab42-9cc7b2433855-tmp-dir\") pod \"simulator-w4-a48099-kserve-7df86f7d-rdlrm\" (UID: \"12c3ad65-a804-429f-ab42-9cc7b2433855\") " pod="e2e-models-e2e-worker-w4-a48099/simulator-w4-a48099-kserve-7df86f7d-rdlrm"
Nov 01 05:16:04.833430 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:04.825022    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12c3ad65-a804-429f-ab42-9cc7b2433855-dshm\") pod \"simulator-w4-a48099-kserve-7df86f7d-rdlrm\" (UID: \"12c3ad65-a804-429f-ab42-9cc7b2433855\") " pod="e2e-models-e2e-worker-w4-a48099/simulator-w4-a48099-kserve-7df86f7d-rdlrm"
Nov 01 05:16:04.833430 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:04.832655    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12c3ad65-a804-429f-ab42-9cc7b2433855-model-cache\") pod \"simulator-w4-a48099-kserve-7df86f7d-rdlrm\" (UID: \"12c3ad65-a804-429f-ab42-9cc7b2433855\") " pod="e2e-models-e2e-worker-w4-a48099/simulator-w4-a48099-kserve-7df86f7d-rdlrm"
Nov 01 05:16:04.833728 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:04.833480    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12c3ad65-a804-429f-ab42-9cc7b2433855-tls-certs\") pod \"simulator-w4-a48099-kserve-7df86f7d-rdlrm\" (UID: \"12c3ad65-a804-429f-ab42-9cc7b2433855\") " pod="e2e-models-e2e-worker-w4-a48099/simulator-w4-a48099-kserve-7df86f7d-rdlrm"
Nov 01 05:16:04.882435 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:04.882402    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kstcx\" (UniqueName: \"kubernetes.io/projected/12c3ad65-a804-429f-ab42-9cc7b2433855-kube-api-access-kstcx\") pod \"simulator-w4-a48099-kserve-7df86f7d-rdlrm\" (UID: \"12c3ad65-a804-429f-ab42-9cc7b2433855\") " pod="e2e-models-e2e-worker-w4-a48099/simulator-w4-a48099-kserve-7df86f7d-rdlrm"
Nov 01 05:16:04.996222 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:04.996147    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-a48099/simulator-w4-a48099-kserve-7df86f7d-rdlrm"
Nov 01 05:16:05.373870 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:05.373794    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-a48099/simulator-w4-a48099-kserve-7df86f7d-rdlrm"]
Nov 01 05:16:05.376630 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:16:05.376604    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod12c3ad65_a804_429f_ab42_9cc7b2433855.slice/crio-57d0cf5226d7ee6af1047d3644a314b19dacb7f6ee0556ec7bb35dfb36979f32 WatchSource:0}: Error finding container 57d0cf5226d7ee6af1047d3644a314b19dacb7f6ee0556ec7bb35dfb36979f32: Status 404 returned error can't find the container with id 57d0cf5226d7ee6af1047d3644a314b19dacb7f6ee0556ec7bb35dfb36979f32
Nov 01 05:16:05.414167 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:05.414140    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-a48099/simulator-w4-a48099-kserve-7df86f7d-rdlrm" event={"ID":"12c3ad65-a804-429f-ab42-9cc7b2433855","Type":"ContainerStarted","Data":"57d0cf5226d7ee6af1047d3644a314b19dacb7f6ee0556ec7bb35dfb36979f32"}
Nov 01 05:16:05.874771 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:05.874738    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:16:06.420931 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:06.420893    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-a48099/simulator-w4-a48099-kserve-7df86f7d-rdlrm" event={"ID":"12c3ad65-a804-429f-ab42-9cc7b2433855","Type":"ContainerStarted","Data":"b2b3617cbf02448c6134ba0af81457476cb4268e819cb1c40c033436439e7178"}
Nov 01 05:16:06.536944 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:06.536891    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-a48099/simulator-w4-a48099-kserve-7df86f7d-rdlrm" podStartSLOduration=2.536875133 podStartE2EDuration="2.536875133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:16:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:16:06.536502886 +0000 UTC m=+744.607299606" watchObservedRunningTime="2026-11-01 05:16:06.536875133 +0000 UTC m=+744.607671854"
Nov 01 05:16:06.595417 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:06.595379    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:16:06.996554 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:06.996521    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-a48099/simulator-w4-a48099-kserve-7df86f7d-rdlrm"
Nov 01 05:16:11.082369 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:11.082334    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-3d70ff/simulator-w2-3d70ff-kserve-898c5fb4f-t5bzk"
Nov 01 05:16:11.096473 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:11.096429    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-3d70ff/simulator-w2-3d70ff-kserve-898c5fb4f-t5bzk"
Nov 01 05:16:11.251874 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:11.251840    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:16:11.396585 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:11.396513    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-2ea125-5cc577d6b5-p45rd"
Nov 01 05:16:11.432018 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:11.431947    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:16:11.467671 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:11.467643    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-3d70ff/simulator-w2-3d70ff-kserve-898c5fb4f-t5bzk"
Nov 01 05:16:11.610284 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:11.610252    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:16:14.996627 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:14.996585    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-a48099/simulator-w4-a48099-kserve-7df86f7d-rdlrm"
Nov 01 05:16:15.009557 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:15.009536    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-a48099/simulator-w4-a48099-kserve-7df86f7d-rdlrm"
Nov 01 05:16:15.486532 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:15.486503    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-a48099/simulator-w4-a48099-kserve-7df86f7d-rdlrm"
Nov 01 05:16:16.309415 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:16.309377    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:16:17.154973 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:17.154940    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:16:20.905870 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:20.905836    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:16:21.116432 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:21.116397    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:16:26.008221 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:26.008178    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:16:30.067924 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:16:30.067890    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:16:30.726522 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:30.726478    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:16:35.501519 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:35.501477    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:16:40.243679 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:40.243643    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:16:43.131650 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:43.131616    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:16:43.576714 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:43.576679    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:16:44.191968 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:44.191934    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:16:44.697393 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:44.697366    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:16:44.881353 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:44.881323    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:16:45.323813 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:45.319000    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:16:48.766336 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:48.766306    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:16:48.962890 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:48.962852    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:16:49.168639 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:49.168597    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:16:50.367862 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:50.367817    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:16:50.848532 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:50.848500    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:16:52.066612 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:52.066574    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:16:52.245210 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:52.245175    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:16:52.617529 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:52.606604    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-af8051-5599f954b9-g8vp8"]
Nov 01 05:16:52.620370 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:52.620341    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-af8051-5599f954b9-g8vp8"
Nov 01 05:16:52.663030 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:52.662987    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-af8051-5599f954b9-g8vp8"]
Nov 01 05:16:52.710564 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:52.710535    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-af8051\""
Nov 01 05:16:52.710713 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:52.710569    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-af8051-dockercfg-jpmwv\""
Nov 01 05:16:52.730010 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:52.729986    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc3118cb-d425-4294-a46c-12a554343918-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-af8051-5599f954b9-g8vp8\" (UID: \"bc3118cb-d425-4294-a46c-12a554343918\") " pod="openshift-ingress/payload-processing-e2e-inf-b-af8051-5599f954b9-g8vp8"
Nov 01 05:16:52.730136 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:52.730122    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t2vx\" (UniqueName: \"kubernetes.io/projected/bc3118cb-d425-4294-a46c-12a554343918-kube-api-access-2t2vx\") pod \"payload-processing-e2e-inf-b-af8051-5599f954b9-g8vp8\" (UID: \"bc3118cb-d425-4294-a46c-12a554343918\") " pod="openshift-ingress/payload-processing-e2e-inf-b-af8051-5599f954b9-g8vp8"
Nov 01 05:16:52.831073 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:52.831043    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t2vx\" (UniqueName: \"kubernetes.io/projected/bc3118cb-d425-4294-a46c-12a554343918-kube-api-access-2t2vx\") pod \"payload-processing-e2e-inf-b-af8051-5599f954b9-g8vp8\" (UID: \"bc3118cb-d425-4294-a46c-12a554343918\") " pod="openshift-ingress/payload-processing-e2e-inf-b-af8051-5599f954b9-g8vp8"
Nov 01 05:16:52.831219 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:52.831151    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc3118cb-d425-4294-a46c-12a554343918-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-af8051-5599f954b9-g8vp8\" (UID: \"bc3118cb-d425-4294-a46c-12a554343918\") " pod="openshift-ingress/payload-processing-e2e-inf-b-af8051-5599f954b9-g8vp8"
Nov 01 05:16:52.832192 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:52.832166    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc3118cb-d425-4294-a46c-12a554343918-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-af8051-5599f954b9-g8vp8\" (UID: \"bc3118cb-d425-4294-a46c-12a554343918\") " pod="openshift-ingress/payload-processing-e2e-inf-b-af8051-5599f954b9-g8vp8"
Nov 01 05:16:52.891806 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:52.891748    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t2vx\" (UniqueName: \"kubernetes.io/projected/bc3118cb-d425-4294-a46c-12a554343918-kube-api-access-2t2vx\") pod \"payload-processing-e2e-inf-b-af8051-5599f954b9-g8vp8\" (UID: \"bc3118cb-d425-4294-a46c-12a554343918\") " pod="openshift-ingress/payload-processing-e2e-inf-b-af8051-5599f954b9-g8vp8"
Nov 01 05:16:52.945620 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:52.945593    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-af8051-5599f954b9-g8vp8"
Nov 01 05:16:53.325990 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:53.325955    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-af8051-5599f954b9-g8vp8"]
Nov 01 05:16:53.329606 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:16:53.329579    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc3118cb_d425_4294_a46c_12a554343918.slice/crio-690e934e6ca73de734172cad2cd3cb21be17ae572c7cc68adc351fc2acb80840 WatchSource:0}: Error finding container 690e934e6ca73de734172cad2cd3cb21be17ae572c7cc68adc351fc2acb80840: Status 404 returned error can't find the container with id 690e934e6ca73de734172cad2cd3cb21be17ae572c7cc68adc351fc2acb80840
Nov 01 05:16:53.664212 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:53.664179    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-af8051-5599f954b9-g8vp8" event={"ID":"bc3118cb-d425-4294-a46c-12a554343918","Type":"ContainerStarted","Data":"8b6e6819a781276f924c22ea6a6f6a39e4fbee54ea16d58a4dbb585c5d4a16f2"}
Nov 01 05:16:53.664383 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:53.664233    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-af8051-5599f954b9-g8vp8" event={"ID":"bc3118cb-d425-4294-a46c-12a554343918","Type":"ContainerStarted","Data":"690e934e6ca73de734172cad2cd3cb21be17ae572c7cc68adc351fc2acb80840"}
Nov 01 05:16:53.664383 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:53.664262    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-af8051-5599f954b9-g8vp8"
Nov 01 05:16:53.785467 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:53.785401    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-af8051-5599f954b9-g8vp8" podStartSLOduration=1.7853890510000001 podStartE2EDuration="1.785389051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:16:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:16:53.783882463 +0000 UTC m=+791.854679217" watchObservedRunningTime="2026-11-01 05:16:53.785389051 +0000 UTC m=+791.856185770"
Nov 01 05:16:55.213892 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:55.213861    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:16:57.900577 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:16:57.900541    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:17:00.012260 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:00.012221    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:17:00.078215 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:17:00.078176    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:17:04.671903 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:04.671868    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-af8051-5599f954b9-g8vp8"
Nov 01 05:17:04.846622 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:04.846581    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:17:07.488723 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:07.488695    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:17:09.616611 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:09.616572    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:17:12.323873 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:12.323844    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:17:14.072709 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:14.072677    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-af8051-8f8c87946-qzrqg"]
Nov 01 05:17:14.076561 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:14.076539    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af8051-8f8c87946-qzrqg"
Nov 01 05:17:14.131184 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:14.131160    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-af8051-serving-cert\""
Nov 01 05:17:14.138707 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:14.138682    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlx8k\" (UniqueName: \"kubernetes.io/projected/b535e368-228c-49c3-b014-445acbad9d0d-kube-api-access-tlx8k\") pod \"maas-api-e2e-inf-b-af8051-8f8c87946-qzrqg\" (UID: \"b535e368-228c-49c3-b014-445acbad9d0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af8051-8f8c87946-qzrqg"
Nov 01 05:17:14.138893 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:14.138870    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b535e368-228c-49c3-b014-445acbad9d0d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-af8051-8f8c87946-qzrqg\" (UID: \"b535e368-228c-49c3-b014-445acbad9d0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af8051-8f8c87946-qzrqg"
Nov 01 05:17:14.138999 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:14.138983    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b535e368-228c-49c3-b014-445acbad9d0d-metrics-certs\") pod \"maas-api-e2e-inf-b-af8051-8f8c87946-qzrqg\" (UID: \"b535e368-228c-49c3-b014-445acbad9d0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af8051-8f8c87946-qzrqg"
Nov 01 05:17:14.139132 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:14.139111    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b535e368-228c-49c3-b014-445acbad9d0d-maas-api-tls\") pod \"maas-api-e2e-inf-b-af8051-8f8c87946-qzrqg\" (UID: \"b535e368-228c-49c3-b014-445acbad9d0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af8051-8f8c87946-qzrqg"
Nov 01 05:17:14.182348 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:14.182322    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-af8051-8f8c87946-qzrqg"]
Nov 01 05:17:14.240763 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:14.240728    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlx8k\" (UniqueName: \"kubernetes.io/projected/b535e368-228c-49c3-b014-445acbad9d0d-kube-api-access-tlx8k\") pod \"maas-api-e2e-inf-b-af8051-8f8c87946-qzrqg\" (UID: \"b535e368-228c-49c3-b014-445acbad9d0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af8051-8f8c87946-qzrqg"
Nov 01 05:17:14.240911 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:14.240839    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b535e368-228c-49c3-b014-445acbad9d0d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-af8051-8f8c87946-qzrqg\" (UID: \"b535e368-228c-49c3-b014-445acbad9d0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af8051-8f8c87946-qzrqg"
Nov 01 05:17:14.240911 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:14.240890    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b535e368-228c-49c3-b014-445acbad9d0d-metrics-certs\") pod \"maas-api-e2e-inf-b-af8051-8f8c87946-qzrqg\" (UID: \"b535e368-228c-49c3-b014-445acbad9d0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af8051-8f8c87946-qzrqg"
Nov 01 05:17:14.241033 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:14.240960    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b535e368-228c-49c3-b014-445acbad9d0d-maas-api-tls\") pod \"maas-api-e2e-inf-b-af8051-8f8c87946-qzrqg\" (UID: \"b535e368-228c-49c3-b014-445acbad9d0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af8051-8f8c87946-qzrqg"
Nov 01 05:17:14.241144 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:17:14.241121    5689 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-af8051-serving-cert: secret "maas-api-e2e-inf-b-af8051-serving-cert" not found
Nov 01 05:17:14.241221 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:17:14.241210    5689 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b535e368-228c-49c3-b014-445acbad9d0d-maas-api-tls podName:b535e368-228c-49c3-b014-445acbad9d0d nodeName:}" failed. No retries permitted until 2026-11-01 05:17:14.741188983 +0000 UTC m=+812.811985700 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b535e368-228c-49c3-b014-445acbad9d0d-maas-api-tls") pod "maas-api-e2e-inf-b-af8051-8f8c87946-qzrqg" (UID: "b535e368-228c-49c3-b014-445acbad9d0d") : secret "maas-api-e2e-inf-b-af8051-serving-cert" not found
Nov 01 05:17:14.241612 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:14.241592    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b535e368-228c-49c3-b014-445acbad9d0d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-af8051-8f8c87946-qzrqg\" (UID: \"b535e368-228c-49c3-b014-445acbad9d0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af8051-8f8c87946-qzrqg"
Nov 01 05:17:14.243348 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:14.243326    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b535e368-228c-49c3-b014-445acbad9d0d-metrics-certs\") pod \"maas-api-e2e-inf-b-af8051-8f8c87946-qzrqg\" (UID: \"b535e368-228c-49c3-b014-445acbad9d0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af8051-8f8c87946-qzrqg"
Nov 01 05:17:14.303957 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:14.303931    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:17:14.304325 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:14.304304    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlx8k\" (UniqueName: \"kubernetes.io/projected/b535e368-228c-49c3-b014-445acbad9d0d-kube-api-access-tlx8k\") pod \"maas-api-e2e-inf-b-af8051-8f8c87946-qzrqg\" (UID: \"b535e368-228c-49c3-b014-445acbad9d0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af8051-8f8c87946-qzrqg"
Nov 01 05:17:14.487476 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:14.484443    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:17:14.670215 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:14.670173    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:17:14.745952 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:14.745885    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b535e368-228c-49c3-b014-445acbad9d0d-maas-api-tls\") pod \"maas-api-e2e-inf-b-af8051-8f8c87946-qzrqg\" (UID: \"b535e368-228c-49c3-b014-445acbad9d0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af8051-8f8c87946-qzrqg"
Nov 01 05:17:14.748336 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:14.748309    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b535e368-228c-49c3-b014-445acbad9d0d-maas-api-tls\") pod \"maas-api-e2e-inf-b-af8051-8f8c87946-qzrqg\" (UID: \"b535e368-228c-49c3-b014-445acbad9d0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af8051-8f8c87946-qzrqg"
Nov 01 05:17:15.010359 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:15.010279    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af8051-8f8c87946-qzrqg"
Nov 01 05:17:15.385506 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:15.385280    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-af8051-8f8c87946-qzrqg"]
Nov 01 05:17:15.388294 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:17:15.388266    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb535e368_228c_49c3_b014_445acbad9d0d.slice/crio-0e464f3668bedb6d0def04d35c2dd4f98524ff363ce025a30ebdd047d9c81dc3 WatchSource:0}: Error finding container 0e464f3668bedb6d0def04d35c2dd4f98524ff363ce025a30ebdd047d9c81dc3: Status 404 returned error can't find the container with id 0e464f3668bedb6d0def04d35c2dd4f98524ff363ce025a30ebdd047d9c81dc3
Nov 01 05:17:15.824305 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:15.824272    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af8051-8f8c87946-qzrqg" event={"ID":"b535e368-228c-49c3-b014-445acbad9d0d","Type":"ContainerStarted","Data":"0e464f3668bedb6d0def04d35c2dd4f98524ff363ce025a30ebdd047d9c81dc3"}
Nov 01 05:17:16.834626 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:16.834591    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af8051-8f8c87946-qzrqg" event={"ID":"b535e368-228c-49c3-b014-445acbad9d0d","Type":"ContainerStarted","Data":"5b4eac7be11fda86a4ac565c99e868b23848fbd275883a42eeed09b577f059cf"}
Nov 01 05:17:16.835048 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:16.834645    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af8051-8f8c87946-qzrqg"
Nov 01 05:17:16.952538 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:16.952438    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af8051-8f8c87946-qzrqg" podStartSLOduration=1.983460149 podStartE2EDuration="2.952419207s" totalImagesPullingTime="968.959058ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:17:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:17:16.951370064 +0000 UTC m=+815.022166786" watchObservedRunningTime="2026-11-01 05:17:16.952419207 +0000 UTC m=+815.023215928"
Nov 01 05:17:19.354854 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:19.354794    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:17:19.737174 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:19.737142    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:17:20.235197 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:20.235162    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l"]
Nov 01 05:17:20.238890 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:20.238868    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l"
Nov 01 05:17:20.293676 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:20.293649    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-a12c35-openshift-default-dockercfg-chl9f\""
Nov 01 05:17:20.302360 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:20.302309    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38e52d68-a35a-4079-a0ec-a7848e909f10-istio-token\") pod \"e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l\" (UID: \"38e52d68-a35a-4079-a0ec-a7848e909f10\") " pod="openshift-ingress/e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l"
Nov 01 05:17:20.302527 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:20.302445    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38e52d68-a35a-4079-a0ec-a7848e909f10-credential-socket\") pod \"e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l\" (UID: \"38e52d68-a35a-4079-a0ec-a7848e909f10\") " pod="openshift-ingress/e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l"
Nov 01 05:17:20.302527 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:20.302500    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dftcx\" (UniqueName: \"kubernetes.io/projected/38e52d68-a35a-4079-a0ec-a7848e909f10-kube-api-access-dftcx\") pod \"e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l\" (UID: \"38e52d68-a35a-4079-a0ec-a7848e909f10\") " pod="openshift-ingress/e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l"
Nov 01 05:17:20.302650 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:20.302633    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38e52d68-a35a-4079-a0ec-a7848e909f10-workload-socket\") pod \"e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l\" (UID: \"38e52d68-a35a-4079-a0ec-a7848e909f10\") " pod="openshift-ingress/e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l"
Nov 01 05:17:20.302703 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:20.302659    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38e52d68-a35a-4079-a0ec-a7848e909f10-istio-envoy\") pod \"e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l\" (UID: \"38e52d68-a35a-4079-a0ec-a7848e909f10\") " pod="openshift-ingress/e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l"
Nov 01 05:17:20.302794 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:20.302774    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38e52d68-a35a-4079-a0ec-a7848e909f10-istio-podinfo\") pod \"e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l\" (UID: \"38e52d68-a35a-4079-a0ec-a7848e909f10\") " pod="openshift-ingress/e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l"
Nov 01 05:17:20.302872 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:20.302859    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38e52d68-a35a-4079-a0ec-a7848e909f10-workload-certs\") pod \"e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l\" (UID: \"38e52d68-a35a-4079-a0ec-a7848e909f10\") " pod="openshift-ingress/e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l"
Nov 01 05:17:20.302926 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:20.302880    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38e52d68-a35a-4079-a0ec-a7848e909f10-istio-data\") pod \"e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l\" (UID: \"38e52d68-a35a-4079-a0ec-a7848e909f10\") " pod="openshift-ingress/e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l"
Nov 01 05:17:20.302926 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:20.302899    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38e52d68-a35a-4079-a0ec-a7848e909f10-istiod-ca-cert\") pod \"e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l\" (UID: \"38e52d68-a35a-4079-a0ec-a7848e909f10\") " pod="openshift-ingress/e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l"
Nov 01 05:17:20.343199 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:20.343170    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l"]
Nov 01 05:17:20.361044 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:20.361018    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:17:20.407134 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:20.407090    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38e52d68-a35a-4079-a0ec-a7848e909f10-workload-socket\") pod \"e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l\" (UID: \"38e52d68-a35a-4079-a0ec-a7848e909f10\") " pod="openshift-ingress/e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l"
Nov 01 05:17:20.407134 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:20.407142    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38e52d68-a35a-4079-a0ec-a7848e909f10-istio-envoy\") pod \"e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l\" (UID: \"38e52d68-a35a-4079-a0ec-a7848e909f10\") " pod="openshift-ingress/e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l"
Nov 01 05:17:20.407393 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:20.407183    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38e52d68-a35a-4079-a0ec-a7848e909f10-istio-podinfo\") pod \"e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l\" (UID: \"38e52d68-a35a-4079-a0ec-a7848e909f10\") " pod="openshift-ingress/e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l"
Nov 01 05:17:20.407393 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:20.407236    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38e52d68-a35a-4079-a0ec-a7848e909f10-workload-certs\") pod \"e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l\" (UID: \"38e52d68-a35a-4079-a0ec-a7848e909f10\") " pod="openshift-ingress/e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l"
Nov 01 05:17:20.407393 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:20.407256    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38e52d68-a35a-4079-a0ec-a7848e909f10-istio-data\") pod \"e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l\" (UID: \"38e52d68-a35a-4079-a0ec-a7848e909f10\") " pod="openshift-ingress/e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l"
Nov 01 05:17:20.407393 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:20.407277    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38e52d68-a35a-4079-a0ec-a7848e909f10-istiod-ca-cert\") pod \"e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l\" (UID: \"38e52d68-a35a-4079-a0ec-a7848e909f10\") " pod="openshift-ingress/e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l"
Nov 01 05:17:20.407393 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:20.407321    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38e52d68-a35a-4079-a0ec-a7848e909f10-istio-token\") pod \"e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l\" (UID: \"38e52d68-a35a-4079-a0ec-a7848e909f10\") " pod="openshift-ingress/e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l"
Nov 01 05:17:20.407393 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:20.407390    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38e52d68-a35a-4079-a0ec-a7848e909f10-credential-socket\") pod \"e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l\" (UID: \"38e52d68-a35a-4079-a0ec-a7848e909f10\") " pod="openshift-ingress/e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l"
Nov 01 05:17:20.408802 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:20.408769    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dftcx\" (UniqueName: \"kubernetes.io/projected/38e52d68-a35a-4079-a0ec-a7848e909f10-kube-api-access-dftcx\") pod \"e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l\" (UID: \"38e52d68-a35a-4079-a0ec-a7848e909f10\") " pod="openshift-ingress/e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l"
Nov 01 05:17:20.409664 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:20.409639    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38e52d68-a35a-4079-a0ec-a7848e909f10-workload-socket\") pod \"e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l\" (UID: \"38e52d68-a35a-4079-a0ec-a7848e909f10\") " pod="openshift-ingress/e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l"
Nov 01 05:17:20.420815 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:20.420785    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38e52d68-a35a-4079-a0ec-a7848e909f10-istio-data\") pod \"e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l\" (UID: \"38e52d68-a35a-4079-a0ec-a7848e909f10\") " pod="openshift-ingress/e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l"
Nov 01 05:17:20.430635 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:20.421344    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38e52d68-a35a-4079-a0ec-a7848e909f10-istiod-ca-cert\") pod \"e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l\" (UID: \"38e52d68-a35a-4079-a0ec-a7848e909f10\") " pod="openshift-ingress/e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l"
Nov 01 05:17:20.430879 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:20.421482    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38e52d68-a35a-4079-a0ec-a7848e909f10-workload-certs\") pod \"e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l\" (UID: \"38e52d68-a35a-4079-a0ec-a7848e909f10\") " pod="openshift-ingress/e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l"
Nov 01 05:17:20.431018 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:20.421596    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38e52d68-a35a-4079-a0ec-a7848e909f10-credential-socket\") pod \"e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l\" (UID: \"38e52d68-a35a-4079-a0ec-a7848e909f10\") " pod="openshift-ingress/e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l"
Nov 01 05:17:20.445472 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:20.444522    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38e52d68-a35a-4079-a0ec-a7848e909f10-istio-podinfo\") pod \"e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l\" (UID: \"38e52d68-a35a-4079-a0ec-a7848e909f10\") " pod="openshift-ingress/e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l"
Nov 01 05:17:20.445472 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:20.444731    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38e52d68-a35a-4079-a0ec-a7848e909f10-istio-envoy\") pod \"e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l\" (UID: \"38e52d68-a35a-4079-a0ec-a7848e909f10\") " pod="openshift-ingress/e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l"
Nov 01 05:17:20.473863 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:20.473837    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dftcx\" (UniqueName: \"kubernetes.io/projected/38e52d68-a35a-4079-a0ec-a7848e909f10-kube-api-access-dftcx\") pod \"e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l\" (UID: \"38e52d68-a35a-4079-a0ec-a7848e909f10\") " pod="openshift-ingress/e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l"
Nov 01 05:17:20.481857 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:20.481832    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38e52d68-a35a-4079-a0ec-a7848e909f10-istio-token\") pod \"e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l\" (UID: \"38e52d68-a35a-4079-a0ec-a7848e909f10\") " pod="openshift-ingress/e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l"
Nov 01 05:17:20.551945 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:20.551857    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l"
Nov 01 05:17:20.564628 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:20.564597    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:17:20.981530 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:20.981492    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l"]
Nov 01 05:17:20.984420 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:17:20.984394    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38e52d68_a35a_4079_a0ec_a7848e909f10.slice/crio-44a69671a995b105c454b483e7727722b29d1c5bb7ebded765773f5e4585a627 WatchSource:0}: Error finding container 44a69671a995b105c454b483e7727722b29d1c5bb7ebded765773f5e4585a627: Status 404 returned error can't find the container with id 44a69671a995b105c454b483e7727722b29d1c5bb7ebded765773f5e4585a627
Nov 01 05:17:20.986961 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:20.986924    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:17:20.987067 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:20.986994    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:17:20.987067 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:20.987027    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:17:21.858629 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:21.858594    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l" event={"ID":"38e52d68-a35a-4079-a0ec-a7848e909f10","Type":"ContainerStarted","Data":"110ab3735dddf842e0eafe96c19dbdc54d9ce6b8d1c58e60930afd3c1b94de85"}
Nov 01 05:17:21.859018 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:21.858653    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l" event={"ID":"38e52d68-a35a-4079-a0ec-a7848e909f10","Type":"ContainerStarted","Data":"44a69671a995b105c454b483e7727722b29d1c5bb7ebded765773f5e4585a627"}
Nov 01 05:17:22.028490 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:22.028422    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l" podStartSLOduration=2.028407938 podStartE2EDuration="2.028407938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:17:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:17:22.027081332 +0000 UTC m=+820.097878053" watchObservedRunningTime="2026-11-01 05:17:22.028407938 +0000 UTC m=+820.099204658"
Nov 01 05:17:22.552368 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:22.552337    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l"
Nov 01 05:17:22.557589 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:22.557563    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l"
Nov 01 05:17:22.845303 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:22.845233    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af8051-8f8c87946-qzrqg"
Nov 01 05:17:22.863792 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:22.863767    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l"
Nov 01 05:17:22.864567 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:22.864548    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l"
Nov 01 05:17:24.250100 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:24.250062    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:17:24.861638 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:24.861606    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:17:25.041326 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:25.041286    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:17:25.234336 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:25.234300    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:17:28.105779 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:28.105739    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:17:28.285659 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:28.285625    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:17:29.635228 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:29.627179    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:17:30.069589 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:17:30.069559    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:17:34.390943 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:34.390914    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:17:39.224484 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:39.224416    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:17:39.912721 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:39.912687    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:17:44.017665 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:44.017617    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:17:48.833955 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:48.833903    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:17:49.851735 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:49.851697    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:17:53.851639 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:53.851608    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:17:54.039268 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:54.039233    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:17:54.216394 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:54.216329    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:17:58.243358 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:58.243299    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:17:58.422494 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:58.422440    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:17:58.901960 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:58.901921    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:17:59.710857 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:17:59.710826    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:18:00.119292 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:18:00.119211    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:18:01.344871 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:01.344838    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:18:03.779536 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:03.779499    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:18:08.500090 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:08.500050    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:18:13.402188 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:13.402141    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:18:18.219472 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:18.219411    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:18:21.439241 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:21.439207    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:18:22.224726 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:22.224686    5689 generic.go:356] "Generic (PLEG): container finished" podID="38e52d68-a35a-4079-a0ec-a7848e909f10" containerID="110ab3735dddf842e0eafe96c19dbdc54d9ce6b8d1c58e60930afd3c1b94de85" exitCode=0
Nov 01 05:18:22.224884 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:22.224762    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l" event={"ID":"38e52d68-a35a-4079-a0ec-a7848e909f10","Type":"ContainerDied","Data":"110ab3735dddf842e0eafe96c19dbdc54d9ce6b8d1c58e60930afd3c1b94de85"}
Nov 01 05:18:22.225228 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:22.225209    5689 scope.go:122] "RemoveContainer" containerID="110ab3735dddf842e0eafe96c19dbdc54d9ce6b8d1c58e60930afd3c1b94de85"
Nov 01 05:18:22.226582 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:22.226556    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:18:22.226667 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:22.226620    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:18:22.226667 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:22.226651    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:18:22.863966 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:22.863933    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l"
Nov 01 05:18:22.954480 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:22.952935    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:18:23.232086 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:23.232051    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l" event={"ID":"38e52d68-a35a-4079-a0ec-a7848e909f10","Type":"ContainerStarted","Data":"fc9f28c29a4c48cf5ad0c48415508ebb81ec43e7cc5496562639e939ac500062"}
Nov 01 05:18:23.552541 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:23.552446    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l"
Nov 01 05:18:23.553500 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:23.553478    5689 patch_prober.go:29] interesting pod/e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Nov 01 05:18:23.553625 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:23.553528    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l" podUID="38e52d68-a35a-4079-a0ec-a7848e909f10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Nov 01 05:18:24.552355 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:24.552321    5689 patch_prober.go:29] interesting pod/e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Nov 01 05:18:24.552787 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:24.552372    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l" podUID="38e52d68-a35a-4079-a0ec-a7848e909f10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Nov 01 05:18:25.553218 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:25.553183    5689 patch_prober.go:29] interesting pod/e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Nov 01 05:18:25.553628 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:25.553242    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l" podUID="38e52d68-a35a-4079-a0ec-a7848e909f10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Nov 01 05:18:26.552402 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:26.552365    5689 patch_prober.go:29] interesting pod/e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Nov 01 05:18:26.552591 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:26.552425    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l" podUID="38e52d68-a35a-4079-a0ec-a7848e909f10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Nov 01 05:18:27.552642 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:27.552610    5689 patch_prober.go:29] interesting pod/e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Nov 01 05:18:27.553015 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:27.552669    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l" podUID="38e52d68-a35a-4079-a0ec-a7848e909f10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Nov 01 05:18:27.702209 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:27.702176    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:18:28.556553 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:28.556523    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l"
Nov 01 05:18:28.556925 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:28.556766    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l"
Nov 01 05:18:28.557316 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:28.557299    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l"
Nov 01 05:18:30.067974 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:18:30.067941    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:18:32.516903 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:32.516869    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:18:36.266362 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:36.266326    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:18:37.631299 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:37.631264    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:18:37.816872 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:37.816839    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:18:38.201635 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:38.201599    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8"]
Nov 01 05:18:38.206088 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:38.206063    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8"
Nov 01 05:18:38.252004 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:38.251967    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82550158-2a2b-4dd6-b8e7-8a777954dfd6-workload-certs\") pod \"e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8\" (UID: \"82550158-2a2b-4dd6-b8e7-8a777954dfd6\") " pod="openshift-ingress/e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8"
Nov 01 05:18:38.252181 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:38.252144    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82550158-2a2b-4dd6-b8e7-8a777954dfd6-istio-envoy\") pod \"e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8\" (UID: \"82550158-2a2b-4dd6-b8e7-8a777954dfd6\") " pod="openshift-ingress/e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8"
Nov 01 05:18:38.252262 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:38.252187    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82550158-2a2b-4dd6-b8e7-8a777954dfd6-istio-podinfo\") pod \"e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8\" (UID: \"82550158-2a2b-4dd6-b8e7-8a777954dfd6\") " pod="openshift-ingress/e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8"
Nov 01 05:18:38.252342 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:38.252318    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82550158-2a2b-4dd6-b8e7-8a777954dfd6-istio-token\") pod \"e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8\" (UID: \"82550158-2a2b-4dd6-b8e7-8a777954dfd6\") " pod="openshift-ingress/e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8"
Nov 01 05:18:38.252478 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:38.252395    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82550158-2a2b-4dd6-b8e7-8a777954dfd6-istio-data\") pod \"e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8\" (UID: \"82550158-2a2b-4dd6-b8e7-8a777954dfd6\") " pod="openshift-ingress/e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8"
Nov 01 05:18:38.252478 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:38.252420    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82550158-2a2b-4dd6-b8e7-8a777954dfd6-istiod-ca-cert\") pod \"e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8\" (UID: \"82550158-2a2b-4dd6-b8e7-8a777954dfd6\") " pod="openshift-ingress/e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8"
Nov 01 05:18:38.252608 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:38.252508    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r5rn\" (UniqueName: \"kubernetes.io/projected/82550158-2a2b-4dd6-b8e7-8a777954dfd6-kube-api-access-9r5rn\") pod \"e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8\" (UID: \"82550158-2a2b-4dd6-b8e7-8a777954dfd6\") " pod="openshift-ingress/e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8"
Nov 01 05:18:38.252608 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:38.252565    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82550158-2a2b-4dd6-b8e7-8a777954dfd6-workload-socket\") pod \"e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8\" (UID: \"82550158-2a2b-4dd6-b8e7-8a777954dfd6\") " pod="openshift-ingress/e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8"
Nov 01 05:18:38.252715 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:38.252625    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82550158-2a2b-4dd6-b8e7-8a777954dfd6-credential-socket\") pod \"e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8\" (UID: \"82550158-2a2b-4dd6-b8e7-8a777954dfd6\") " pod="openshift-ingress/e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8"
Nov 01 05:18:38.278797 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:38.278762    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:18:38.280799 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:38.280771    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8"]
Nov 01 05:18:38.318185 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:38.318162    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-2fd4c9-openshift-default-dockercfg-j7f4z\""
Nov 01 05:18:38.372227 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:38.372190    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82550158-2a2b-4dd6-b8e7-8a777954dfd6-credential-socket\") pod \"e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8\" (UID: \"82550158-2a2b-4dd6-b8e7-8a777954dfd6\") " pod="openshift-ingress/e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8"
Nov 01 05:18:38.372494 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:38.372473    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82550158-2a2b-4dd6-b8e7-8a777954dfd6-workload-certs\") pod \"e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8\" (UID: \"82550158-2a2b-4dd6-b8e7-8a777954dfd6\") " pod="openshift-ingress/e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8"
Nov 01 05:18:38.372717 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:38.372699    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82550158-2a2b-4dd6-b8e7-8a777954dfd6-istio-envoy\") pod \"e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8\" (UID: \"82550158-2a2b-4dd6-b8e7-8a777954dfd6\") " pod="openshift-ingress/e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8"
Nov 01 05:18:38.373609 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:38.373559    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82550158-2a2b-4dd6-b8e7-8a777954dfd6-istio-podinfo\") pod \"e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8\" (UID: \"82550158-2a2b-4dd6-b8e7-8a777954dfd6\") " pod="openshift-ingress/e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8"
Nov 01 05:18:38.373741 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:38.373636    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82550158-2a2b-4dd6-b8e7-8a777954dfd6-istio-token\") pod \"e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8\" (UID: \"82550158-2a2b-4dd6-b8e7-8a777954dfd6\") " pod="openshift-ingress/e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8"
Nov 01 05:18:38.373741 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:38.373717    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82550158-2a2b-4dd6-b8e7-8a777954dfd6-istio-data\") pod \"e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8\" (UID: \"82550158-2a2b-4dd6-b8e7-8a777954dfd6\") " pod="openshift-ingress/e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8"
Nov 01 05:18:38.373859 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:38.373742    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82550158-2a2b-4dd6-b8e7-8a777954dfd6-istiod-ca-cert\") pod \"e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8\" (UID: \"82550158-2a2b-4dd6-b8e7-8a777954dfd6\") " pod="openshift-ingress/e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8"
Nov 01 05:18:38.373859 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:38.373769    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r5rn\" (UniqueName: \"kubernetes.io/projected/82550158-2a2b-4dd6-b8e7-8a777954dfd6-kube-api-access-9r5rn\") pod \"e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8\" (UID: \"82550158-2a2b-4dd6-b8e7-8a777954dfd6\") " pod="openshift-ingress/e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8"
Nov 01 05:18:38.373859 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:38.373798    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82550158-2a2b-4dd6-b8e7-8a777954dfd6-workload-socket\") pod \"e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8\" (UID: \"82550158-2a2b-4dd6-b8e7-8a777954dfd6\") " pod="openshift-ingress/e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8"
Nov 01 05:18:38.375110 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:38.374156    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82550158-2a2b-4dd6-b8e7-8a777954dfd6-workload-socket\") pod \"e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8\" (UID: \"82550158-2a2b-4dd6-b8e7-8a777954dfd6\") " pod="openshift-ingress/e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8"
Nov 01 05:18:38.375110 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:38.373173    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82550158-2a2b-4dd6-b8e7-8a777954dfd6-credential-socket\") pod \"e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8\" (UID: \"82550158-2a2b-4dd6-b8e7-8a777954dfd6\") " pod="openshift-ingress/e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8"
Nov 01 05:18:38.375110 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:38.374404    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82550158-2a2b-4dd6-b8e7-8a777954dfd6-istio-data\") pod \"e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8\" (UID: \"82550158-2a2b-4dd6-b8e7-8a777954dfd6\") " pod="openshift-ingress/e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8"
Nov 01 05:18:38.375110 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:38.373387    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82550158-2a2b-4dd6-b8e7-8a777954dfd6-workload-certs\") pod \"e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8\" (UID: \"82550158-2a2b-4dd6-b8e7-8a777954dfd6\") " pod="openshift-ingress/e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8"
Nov 01 05:18:38.375110 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:38.375067    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82550158-2a2b-4dd6-b8e7-8a777954dfd6-istiod-ca-cert\") pod \"e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8\" (UID: \"82550158-2a2b-4dd6-b8e7-8a777954dfd6\") " pod="openshift-ingress/e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8"
Nov 01 05:18:38.389270 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:38.389243    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82550158-2a2b-4dd6-b8e7-8a777954dfd6-istio-podinfo\") pod \"e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8\" (UID: \"82550158-2a2b-4dd6-b8e7-8a777954dfd6\") " pod="openshift-ingress/e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8"
Nov 01 05:18:38.401493 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:38.393909    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82550158-2a2b-4dd6-b8e7-8a777954dfd6-istio-envoy\") pod \"e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8\" (UID: \"82550158-2a2b-4dd6-b8e7-8a777954dfd6\") " pod="openshift-ingress/e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8"
Nov 01 05:18:38.436889 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:38.436854    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82550158-2a2b-4dd6-b8e7-8a777954dfd6-istio-token\") pod \"e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8\" (UID: \"82550158-2a2b-4dd6-b8e7-8a777954dfd6\") " pod="openshift-ingress/e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8"
Nov 01 05:18:38.437075 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:38.437052    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r5rn\" (UniqueName: \"kubernetes.io/projected/82550158-2a2b-4dd6-b8e7-8a777954dfd6-kube-api-access-9r5rn\") pod \"e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8\" (UID: \"82550158-2a2b-4dd6-b8e7-8a777954dfd6\") " pod="openshift-ingress/e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8"
Nov 01 05:18:38.457792 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:38.457724    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:18:38.519328 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:38.519301    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8"
Nov 01 05:18:38.630978 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:38.630774    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:18:39.003040 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:39.003010    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8"]
Nov 01 05:18:39.006220 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:18:39.006195    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82550158_2a2b_4dd6_b8e7_8a777954dfd6.slice/crio-d4ce87b1a64dc2a740d826d1442e214f435a78437a81c03db5a8e08461285eb6 WatchSource:0}: Error finding container d4ce87b1a64dc2a740d826d1442e214f435a78437a81c03db5a8e08461285eb6: Status 404 returned error can't find the container with id d4ce87b1a64dc2a740d826d1442e214f435a78437a81c03db5a8e08461285eb6
Nov 01 05:18:39.008494 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:39.008444    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:18:39.008558 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:39.008525    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:18:39.008596 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:39.008555    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:18:39.318786 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:39.318701    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8" event={"ID":"82550158-2a2b-4dd6-b8e7-8a777954dfd6","Type":"ContainerStarted","Data":"68b67a65e0852248333ebf340d765a36b08644b323b5e99575a2a8409bc5715b"}
Nov 01 05:18:39.318786 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:39.318754    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8" event={"ID":"82550158-2a2b-4dd6-b8e7-8a777954dfd6","Type":"ContainerStarted","Data":"d4ce87b1a64dc2a740d826d1442e214f435a78437a81c03db5a8e08461285eb6"}
Nov 01 05:18:39.415246 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:39.415215    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:18:39.438656 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:39.438592    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8" podStartSLOduration=1.438531609 podStartE2EDuration="1.438531609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:18:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:18:39.437798309 +0000 UTC m=+897.508595030" watchObservedRunningTime="2026-11-01 05:18:39.438531609 +0000 UTC m=+897.509328330"
Nov 01 05:18:39.519405 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:39.519358    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8"
Nov 01 05:18:40.525630 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:40.525603    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8"
Nov 01 05:18:41.329560 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:41.329531    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8"
Nov 01 05:18:41.330575 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:41.330556    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8"
Nov 01 05:18:42.278402 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:42.278374    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:18:42.460800 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:42.460771    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:18:42.631088 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:42.631007    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:18:42.806940 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:42.806906    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:18:46.788932 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:46.788900    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:18:46.967040 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:46.967007    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:18:47.272664 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:47.272628    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-2fd4c9-7b49b85457-4pklk"]
Nov 01 05:18:47.280040 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:47.280007    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-2fd4c9-7b49b85457-4pklk"
Nov 01 05:18:47.331561 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:47.331515    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-2fd4c9-7b49b85457-4pklk"]
Nov 01 05:18:47.373611 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:47.373581    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzzgh\" (UniqueName: \"kubernetes.io/projected/cde94b55-8dd3-426e-aafb-670ec1c4ee7d-kube-api-access-hzzgh\") pod \"payload-processing-e2e-extmod-b-2fd4c9-7b49b85457-4pklk\" (UID: \"cde94b55-8dd3-426e-aafb-670ec1c4ee7d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-2fd4c9-7b49b85457-4pklk"
Nov 01 05:18:47.373744 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:47.373617    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cde94b55-8dd3-426e-aafb-670ec1c4ee7d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-2fd4c9-7b49b85457-4pklk\" (UID: \"cde94b55-8dd3-426e-aafb-670ec1c4ee7d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-2fd4c9-7b49b85457-4pklk"
Nov 01 05:18:47.378791 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:47.378771    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-2fd4c9\""
Nov 01 05:18:47.379048 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:47.379032    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-2fd4c9-dockercfg-pxh9b\""
Nov 01 05:18:47.474759 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:47.474732    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzzgh\" (UniqueName: \"kubernetes.io/projected/cde94b55-8dd3-426e-aafb-670ec1c4ee7d-kube-api-access-hzzgh\") pod \"payload-processing-e2e-extmod-b-2fd4c9-7b49b85457-4pklk\" (UID: \"cde94b55-8dd3-426e-aafb-670ec1c4ee7d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-2fd4c9-7b49b85457-4pklk"
Nov 01 05:18:47.474937 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:47.474766    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cde94b55-8dd3-426e-aafb-670ec1c4ee7d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-2fd4c9-7b49b85457-4pklk\" (UID: \"cde94b55-8dd3-426e-aafb-670ec1c4ee7d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-2fd4c9-7b49b85457-4pklk"
Nov 01 05:18:47.475440 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:47.475420    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cde94b55-8dd3-426e-aafb-670ec1c4ee7d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-2fd4c9-7b49b85457-4pklk\" (UID: \"cde94b55-8dd3-426e-aafb-670ec1c4ee7d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-2fd4c9-7b49b85457-4pklk"
Nov 01 05:18:47.535717 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:47.535652    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzzgh\" (UniqueName: \"kubernetes.io/projected/cde94b55-8dd3-426e-aafb-670ec1c4ee7d-kube-api-access-hzzgh\") pod \"payload-processing-e2e-extmod-b-2fd4c9-7b49b85457-4pklk\" (UID: \"cde94b55-8dd3-426e-aafb-670ec1c4ee7d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-2fd4c9-7b49b85457-4pklk"
Nov 01 05:18:47.595766 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:47.595736    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-2fd4c9-7b49b85457-4pklk"
Nov 01 05:18:47.612728 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:47.612486    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:18:47.998381 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:47.998350    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-2fd4c9-7b49b85457-4pklk"]
Nov 01 05:18:48.001426 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:18:48.001399    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcde94b55_8dd3_426e_aafb_670ec1c4ee7d.slice/crio-aa26aed818575c331027e5e16d087444ede2ea43850b4f6484a10eb95f61fd54 WatchSource:0}: Error finding container aa26aed818575c331027e5e16d087444ede2ea43850b4f6484a10eb95f61fd54: Status 404 returned error can't find the container with id aa26aed818575c331027e5e16d087444ede2ea43850b4f6484a10eb95f61fd54
Nov 01 05:18:48.364614 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:48.364536    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-2fd4c9-7b49b85457-4pklk" event={"ID":"cde94b55-8dd3-426e-aafb-670ec1c4ee7d","Type":"ContainerStarted","Data":"56a1f56199b1f824c2eddd37627c440812f3f69e481bf8cb839ee56895456655"}
Nov 01 05:18:48.364614 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:48.364584    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-2fd4c9-7b49b85457-4pklk" event={"ID":"cde94b55-8dd3-426e-aafb-670ec1c4ee7d","Type":"ContainerStarted","Data":"aa26aed818575c331027e5e16d087444ede2ea43850b4f6484a10eb95f61fd54"}
Nov 01 05:18:48.364789 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:48.364669    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-2fd4c9-7b49b85457-4pklk"
Nov 01 05:18:48.483570 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:48.483523    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-2fd4c9-7b49b85457-4pklk" podStartSLOduration=1.483507745 podStartE2EDuration="1.483507745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:18:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:18:48.480587678 +0000 UTC m=+906.551384398" watchObservedRunningTime="2026-11-01 05:18:48.483507745 +0000 UTC m=+906.554304466"
Nov 01 05:18:52.486958 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:52.486874    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:18:57.322906 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:57.322871    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:18:58.592877 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:58.592828    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:18:59.372612 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:18:59.372587    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-2fd4c9-7b49b85457-4pklk"
Nov 01 05:19:00.068964 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:19:00.068934    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:19:01.584086 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:19:01.584053    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:19:02.198712 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:19:02.198673    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:19:05.179246 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:19:05.179198    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:19:06.996583 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:19:06.996539    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:19:07.251535 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:19:07.251440    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:19:07.427354 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:19:07.427324    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:19:11.716473 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:19:11.716405    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:19:16.227319 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:19:16.227282    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-2fd4c9-6674594479-74lqf"]
Nov 01 05:19:16.234182 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:19:16.234147    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2fd4c9-6674594479-74lqf"
Nov 01 05:19:16.284830 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:19:16.284780    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fe0ed2a-6b37-4a3c-bcaf-21afae50cafb-metrics-certs\") pod \"maas-api-e2e-extmod-b-2fd4c9-6674594479-74lqf\" (UID: \"9fe0ed2a-6b37-4a3c-bcaf-21afae50cafb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2fd4c9-6674594479-74lqf"
Nov 01 05:19:16.284830 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:19:16.284828    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fe0ed2a-6b37-4a3c-bcaf-21afae50cafb-maas-api-tls\") pod \"maas-api-e2e-extmod-b-2fd4c9-6674594479-74lqf\" (UID: \"9fe0ed2a-6b37-4a3c-bcaf-21afae50cafb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2fd4c9-6674594479-74lqf"
Nov 01 05:19:16.285051 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:19:16.284915    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fe0ed2a-6b37-4a3c-bcaf-21afae50cafb-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-2fd4c9-6674594479-74lqf\" (UID: \"9fe0ed2a-6b37-4a3c-bcaf-21afae50cafb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2fd4c9-6674594479-74lqf"
Nov 01 05:19:16.285051 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:19:16.284957    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq84z\" (UniqueName: \"kubernetes.io/projected/9fe0ed2a-6b37-4a3c-bcaf-21afae50cafb-kube-api-access-dq84z\") pod \"maas-api-e2e-extmod-b-2fd4c9-6674594479-74lqf\" (UID: \"9fe0ed2a-6b37-4a3c-bcaf-21afae50cafb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2fd4c9-6674594479-74lqf"
Nov 01 05:19:16.287244 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:19:16.287210    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-2fd4c9-6674594479-74lqf"]
Nov 01 05:19:16.334498 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:19:16.334470    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-2fd4c9-serving-cert\""
Nov 01 05:19:16.386796 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:19:16.386757    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fe0ed2a-6b37-4a3c-bcaf-21afae50cafb-metrics-certs\") pod \"maas-api-e2e-extmod-b-2fd4c9-6674594479-74lqf\" (UID: \"9fe0ed2a-6b37-4a3c-bcaf-21afae50cafb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2fd4c9-6674594479-74lqf"
Nov 01 05:19:16.386939 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:19:16.386822    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fe0ed2a-6b37-4a3c-bcaf-21afae50cafb-maas-api-tls\") pod \"maas-api-e2e-extmod-b-2fd4c9-6674594479-74lqf\" (UID: \"9fe0ed2a-6b37-4a3c-bcaf-21afae50cafb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2fd4c9-6674594479-74lqf"
Nov 01 05:19:16.386939 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:19:16.386858    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fe0ed2a-6b37-4a3c-bcaf-21afae50cafb-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-2fd4c9-6674594479-74lqf\" (UID: \"9fe0ed2a-6b37-4a3c-bcaf-21afae50cafb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2fd4c9-6674594479-74lqf"
Nov 01 05:19:16.387023 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:19:16.386938    5689 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-2fd4c9-serving-cert: secret "maas-api-e2e-extmod-b-2fd4c9-serving-cert" not found
Nov 01 05:19:16.387023 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:19:16.386973    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq84z\" (UniqueName: \"kubernetes.io/projected/9fe0ed2a-6b37-4a3c-bcaf-21afae50cafb-kube-api-access-dq84z\") pod \"maas-api-e2e-extmod-b-2fd4c9-6674594479-74lqf\" (UID: \"9fe0ed2a-6b37-4a3c-bcaf-21afae50cafb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2fd4c9-6674594479-74lqf"
Nov 01 05:19:16.387023 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:19:16.386997    5689 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fe0ed2a-6b37-4a3c-bcaf-21afae50cafb-maas-api-tls podName:9fe0ed2a-6b37-4a3c-bcaf-21afae50cafb nodeName:}" failed. No retries permitted until 2026-11-01 05:19:16.886982491 +0000 UTC m=+934.957779192 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9fe0ed2a-6b37-4a3c-bcaf-21afae50cafb-maas-api-tls") pod "maas-api-e2e-extmod-b-2fd4c9-6674594479-74lqf" (UID: "9fe0ed2a-6b37-4a3c-bcaf-21afae50cafb") : secret "maas-api-e2e-extmod-b-2fd4c9-serving-cert" not found
Nov 01 05:19:16.387670 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:19:16.387606    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fe0ed2a-6b37-4a3c-bcaf-21afae50cafb-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-2fd4c9-6674594479-74lqf\" (UID: \"9fe0ed2a-6b37-4a3c-bcaf-21afae50cafb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2fd4c9-6674594479-74lqf"
Nov 01 05:19:16.389276 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:19:16.389256    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fe0ed2a-6b37-4a3c-bcaf-21afae50cafb-metrics-certs\") pod \"maas-api-e2e-extmod-b-2fd4c9-6674594479-74lqf\" (UID: \"9fe0ed2a-6b37-4a3c-bcaf-21afae50cafb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2fd4c9-6674594479-74lqf"
Nov 01 05:19:16.448516 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:19:16.448489    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq84z\" (UniqueName: \"kubernetes.io/projected/9fe0ed2a-6b37-4a3c-bcaf-21afae50cafb-kube-api-access-dq84z\") pod \"maas-api-e2e-extmod-b-2fd4c9-6674594479-74lqf\" (UID: \"9fe0ed2a-6b37-4a3c-bcaf-21afae50cafb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2fd4c9-6674594479-74lqf"
Nov 01 05:19:16.499130 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:19:16.499043    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:19:16.676665 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:19:16.676624    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:19:16.884557 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:19:16.884428    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:19:16.893887 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:19:16.893864    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fe0ed2a-6b37-4a3c-bcaf-21afae50cafb-maas-api-tls\") pod \"maas-api-e2e-extmod-b-2fd4c9-6674594479-74lqf\" (UID: \"9fe0ed2a-6b37-4a3c-bcaf-21afae50cafb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2fd4c9-6674594479-74lqf"
Nov 01 05:19:16.896413 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:19:16.896388    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fe0ed2a-6b37-4a3c-bcaf-21afae50cafb-maas-api-tls\") pod \"maas-api-e2e-extmod-b-2fd4c9-6674594479-74lqf\" (UID: \"9fe0ed2a-6b37-4a3c-bcaf-21afae50cafb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2fd4c9-6674594479-74lqf"
Nov 01 05:19:17.149143 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:19:17.149114    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2fd4c9-6674594479-74lqf"
Nov 01 05:19:17.526253 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:19:17.526172    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-2fd4c9-6674594479-74lqf"]
Nov 01 05:19:17.529325 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:19:17.529298    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fe0ed2a_6b37_4a3c_bcaf_21afae50cafb.slice/crio-3088623c340b3fd5b3c28fc1a308835c714bb2a12679875a5bf7128da5207f31 WatchSource:0}: Error finding container 3088623c340b3fd5b3c28fc1a308835c714bb2a12679875a5bf7128da5207f31: Status 404 returned error can't find the container with id 3088623c340b3fd5b3c28fc1a308835c714bb2a12679875a5bf7128da5207f31
Nov 01 05:19:17.560731 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:19:17.560702    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2fd4c9-6674594479-74lqf" event={"ID":"9fe0ed2a-6b37-4a3c-bcaf-21afae50cafb","Type":"ContainerStarted","Data":"3088623c340b3fd5b3c28fc1a308835c714bb2a12679875a5bf7128da5207f31"}
Nov 01 05:19:18.567993 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:19:18.567959    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2fd4c9-6674594479-74lqf" event={"ID":"9fe0ed2a-6b37-4a3c-bcaf-21afae50cafb","Type":"ContainerStarted","Data":"8d5493454e66dbf81e30cabe5c4d3b0edc5be13c82b7d86c6483f7d6ced1b769"}
Nov 01 05:19:18.568362 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:19:18.568027    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2fd4c9-6674594479-74lqf"
Nov 01 05:19:18.695586 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:19:18.695486    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2fd4c9-6674594479-74lqf" podStartSLOduration=1.823564562 podStartE2EDuration="2.695445925s" totalImagesPullingTime="871.881363ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:19:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:19:18.694649944 +0000 UTC m=+936.765446688" watchObservedRunningTime="2026-11-01 05:19:18.695445925 +0000 UTC m=+936.766242644"
Nov 01 05:19:21.401160 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:19:21.401122    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:19:22.870397 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:19:22.870366    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:19:23.058788 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:19:23.058758    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:19:24.578718 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:19:24.578692    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2fd4c9-6674594479-74lqf"
Nov 01 05:19:26.350376 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:19:26.348515    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:19:30.068793 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:19:30.068753    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:19:31.167799 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:19:31.167768    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:19:35.871469 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:19:35.871421    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:19:35.922020 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:19:35.921990    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:19:40.743051 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:19:40.742961    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:19:41.447139 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:19:41.447100    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:19:45.573028 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:19:45.572981    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:19:46.753096 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:19:46.753059    5689 generic.go:356] "Generic (PLEG): container finished" podID="82550158-2a2b-4dd6-b8e7-8a777954dfd6" containerID="68b67a65e0852248333ebf340d765a36b08644b323b5e99575a2a8409bc5715b" exitCode=0
Nov 01 05:19:46.753604 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:19:46.753128    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8" event={"ID":"82550158-2a2b-4dd6-b8e7-8a777954dfd6","Type":"ContainerDied","Data":"68b67a65e0852248333ebf340d765a36b08644b323b5e99575a2a8409bc5715b"}
Nov 01 05:19:46.753604 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:19:46.753540    5689 scope.go:122] "RemoveContainer" containerID="68b67a65e0852248333ebf340d765a36b08644b323b5e99575a2a8409bc5715b"
Nov 01 05:19:46.754934 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:19:46.754906    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:19:46.754997 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:19:46.754973    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:19:46.755032 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:19:46.755008    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:19:47.760292 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:19:47.760257    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8" event={"ID":"82550158-2a2b-4dd6-b8e7-8a777954dfd6","Type":"ContainerStarted","Data":"09e84abd09aae2ef6adb280141f2c35b9f92034c18c685d8c187436c6b424ba7"}
Nov 01 05:19:48.520425 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:19:48.520391    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8"
Nov 01 05:19:48.520601 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:19:48.520509    5689 patch_prober.go:29] interesting pod/e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 01 05:19:48.520601 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:19:48.520551    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8" podUID="82550158-2a2b-4dd6-b8e7-8a777954dfd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 01 05:19:49.519581 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:19:49.519549    5689 patch_prober.go:29] interesting pod/e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 01 05:19:49.519989 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:19:49.519602    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8" podUID="82550158-2a2b-4dd6-b8e7-8a777954dfd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 01 05:19:50.335687 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:19:50.335648    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:19:50.520216 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:19:50.520175    5689 patch_prober.go:29] interesting pod/e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 01 05:19:50.520723 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:19:50.520230    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8" podUID="82550158-2a2b-4dd6-b8e7-8a777954dfd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 01 05:19:51.524478 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:19:51.520788    5689 patch_prober.go:29] interesting pod/e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 01 05:19:51.524478 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:19:51.520847    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8" podUID="82550158-2a2b-4dd6-b8e7-8a777954dfd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 01 05:19:52.520101 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:19:52.520064    5689 patch_prober.go:29] interesting pod/e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 01 05:19:52.520398 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:19:52.520368    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8" podUID="82550158-2a2b-4dd6-b8e7-8a777954dfd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 01 05:19:53.525560 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:19:53.525531    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8"
Nov 01 05:19:53.526014 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:19:53.525832    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8"
Nov 01 05:19:53.526568 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:19:53.526552    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8"
Nov 01 05:19:55.136098 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:19:55.136046    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:19:58.930329 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:19:58.930296    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:19:59.903766 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:19:59.903733    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:20:00.068400 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:20:00.068368    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:20:02.867803 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:02.867750    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:20:03.476049 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:03.475962    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:20:03.681341 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:03.681302    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:20:03.719244 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:03.719205    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:20:03.920476 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:03.918100    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:20:05.018918 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:05.018885    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:20:05.285769 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:05.285683    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:20:05.490802 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:05.490767    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:20:05.776897 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:05.776846    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:20:05.966284 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:05.966250    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:20:06.999841 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:06.999806    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:20:07.185989 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:07.185855    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:20:07.363718 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:07.363641    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:20:07.537230 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:07.537194    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:20:08.339688 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:08.339656    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-p6p56"]
Nov 01 05:20:08.343739 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:08.343718    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-p6p56"
Nov 01 05:20:08.398364 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:08.398338    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-a48099-openshift-default-dockercfg-8mhxp\""
Nov 01 05:20:08.443065 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:08.443036    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc051a97-b093-4019-957f-63b017c153a5-workload-socket\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-p6p56\" (UID: \"fc051a97-b093-4019-957f-63b017c153a5\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-p6p56"
Nov 01 05:20:08.443203 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:08.443095    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc051a97-b093-4019-957f-63b017c153a5-credential-socket\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-p6p56\" (UID: \"fc051a97-b093-4019-957f-63b017c153a5\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-p6p56"
Nov 01 05:20:08.443203 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:08.443194    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc051a97-b093-4019-957f-63b017c153a5-istio-data\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-p6p56\" (UID: \"fc051a97-b093-4019-957f-63b017c153a5\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-p6p56"
Nov 01 05:20:08.443283 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:08.443225    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc051a97-b093-4019-957f-63b017c153a5-istio-podinfo\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-p6p56\" (UID: \"fc051a97-b093-4019-957f-63b017c153a5\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-p6p56"
Nov 01 05:20:08.443380 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:08.443359    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc051a97-b093-4019-957f-63b017c153a5-istio-token\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-p6p56\" (UID: \"fc051a97-b093-4019-957f-63b017c153a5\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-p6p56"
Nov 01 05:20:08.443512 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:08.443494    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc051a97-b093-4019-957f-63b017c153a5-workload-certs\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-p6p56\" (UID: \"fc051a97-b093-4019-957f-63b017c153a5\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-p6p56"
Nov 01 05:20:08.443553 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:08.443518    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc051a97-b093-4019-957f-63b017c153a5-istiod-ca-cert\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-p6p56\" (UID: \"fc051a97-b093-4019-957f-63b017c153a5\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-p6p56"
Nov 01 05:20:08.443553 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:08.443543    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc051a97-b093-4019-957f-63b017c153a5-istio-envoy\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-p6p56\" (UID: \"fc051a97-b093-4019-957f-63b017c153a5\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-p6p56"
Nov 01 05:20:08.443709 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:08.443681    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjmcs\" (UniqueName: \"kubernetes.io/projected/fc051a97-b093-4019-957f-63b017c153a5-kube-api-access-zjmcs\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-p6p56\" (UID: \"fc051a97-b093-4019-957f-63b017c153a5\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-p6p56"
Nov 01 05:20:08.449860 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:08.449835    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-p6p56"]
Nov 01 05:20:08.544429 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:08.544390    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc051a97-b093-4019-957f-63b017c153a5-istio-envoy\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-p6p56\" (UID: \"fc051a97-b093-4019-957f-63b017c153a5\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-p6p56"
Nov 01 05:20:08.544601 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:08.544514    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjmcs\" (UniqueName: \"kubernetes.io/projected/fc051a97-b093-4019-957f-63b017c153a5-kube-api-access-zjmcs\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-p6p56\" (UID: \"fc051a97-b093-4019-957f-63b017c153a5\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-p6p56"
Nov 01 05:20:08.544601 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:08.544573    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc051a97-b093-4019-957f-63b017c153a5-workload-socket\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-p6p56\" (UID: \"fc051a97-b093-4019-957f-63b017c153a5\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-p6p56"
Nov 01 05:20:08.544682 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:08.544629    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc051a97-b093-4019-957f-63b017c153a5-credential-socket\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-p6p56\" (UID: \"fc051a97-b093-4019-957f-63b017c153a5\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-p6p56"
Nov 01 05:20:08.544731 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:08.544679    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc051a97-b093-4019-957f-63b017c153a5-istio-data\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-p6p56\" (UID: \"fc051a97-b093-4019-957f-63b017c153a5\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-p6p56"
Nov 01 05:20:08.544731 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:08.544708    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc051a97-b093-4019-957f-63b017c153a5-istio-podinfo\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-p6p56\" (UID: \"fc051a97-b093-4019-957f-63b017c153a5\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-p6p56"
Nov 01 05:20:08.544829 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:08.544762    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc051a97-b093-4019-957f-63b017c153a5-istio-token\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-p6p56\" (UID: \"fc051a97-b093-4019-957f-63b017c153a5\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-p6p56"
Nov 01 05:20:08.544885 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:08.544868    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc051a97-b093-4019-957f-63b017c153a5-workload-certs\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-p6p56\" (UID: \"fc051a97-b093-4019-957f-63b017c153a5\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-p6p56"
Nov 01 05:20:08.544943 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:08.544892    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc051a97-b093-4019-957f-63b017c153a5-istiod-ca-cert\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-p6p56\" (UID: \"fc051a97-b093-4019-957f-63b017c153a5\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-p6p56"
Nov 01 05:20:08.544943 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:08.544903    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc051a97-b093-4019-957f-63b017c153a5-workload-socket\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-p6p56\" (UID: \"fc051a97-b093-4019-957f-63b017c153a5\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-p6p56"
Nov 01 05:20:08.544943 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:08.544934    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc051a97-b093-4019-957f-63b017c153a5-credential-socket\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-p6p56\" (UID: \"fc051a97-b093-4019-957f-63b017c153a5\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-p6p56"
Nov 01 05:20:08.545101 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:08.545062    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc051a97-b093-4019-957f-63b017c153a5-istio-data\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-p6p56\" (UID: \"fc051a97-b093-4019-957f-63b017c153a5\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-p6p56"
Nov 01 05:20:08.545274 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:08.545252    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc051a97-b093-4019-957f-63b017c153a5-workload-certs\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-p6p56\" (UID: \"fc051a97-b093-4019-957f-63b017c153a5\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-p6p56"
Nov 01 05:20:08.545578 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:08.545556    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc051a97-b093-4019-957f-63b017c153a5-istiod-ca-cert\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-p6p56\" (UID: \"fc051a97-b093-4019-957f-63b017c153a5\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-p6p56"
Nov 01 05:20:08.546755 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:08.546728    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc051a97-b093-4019-957f-63b017c153a5-istio-envoy\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-p6p56\" (UID: \"fc051a97-b093-4019-957f-63b017c153a5\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-p6p56"
Nov 01 05:20:08.547342 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:08.547323    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc051a97-b093-4019-957f-63b017c153a5-istio-podinfo\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-p6p56\" (UID: \"fc051a97-b093-4019-957f-63b017c153a5\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-p6p56"
Nov 01 05:20:08.604613 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:08.604551    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc051a97-b093-4019-957f-63b017c153a5-istio-token\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-p6p56\" (UID: \"fc051a97-b093-4019-957f-63b017c153a5\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-p6p56"
Nov 01 05:20:08.604745 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:08.604728    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjmcs\" (UniqueName: \"kubernetes.io/projected/fc051a97-b093-4019-957f-63b017c153a5-kube-api-access-zjmcs\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-p6p56\" (UID: \"fc051a97-b093-4019-957f-63b017c153a5\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-p6p56"
Nov 01 05:20:08.657717 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:08.657696    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-p6p56"
Nov 01 05:20:09.083897 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:09.083870    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-p6p56"]
Nov 01 05:20:09.086610 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:20:09.086581    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc051a97_b093_4019_957f_63b017c153a5.slice/crio-05ef50fb96737c173b4019a520c10afb4882dd61f54cf25a25a633ab316de8ce WatchSource:0}: Error finding container 05ef50fb96737c173b4019a520c10afb4882dd61f54cf25a25a633ab316de8ce: Status 404 returned error can't find the container with id 05ef50fb96737c173b4019a520c10afb4882dd61f54cf25a25a633ab316de8ce
Nov 01 05:20:09.088900 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:09.088867    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:20:09.088987 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:09.088933    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:20:09.088987 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:09.088963    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:20:09.932378 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:09.932341    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-p6p56" event={"ID":"fc051a97-b093-4019-957f-63b017c153a5","Type":"ContainerStarted","Data":"5803d671cf007499e141b4e69ac378d09c84b60af4294f801da5184a5e76e068"}
Nov 01 05:20:09.932820 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:09.932395    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-p6p56" event={"ID":"fc051a97-b093-4019-957f-63b017c153a5","Type":"ContainerStarted","Data":"05ef50fb96737c173b4019a520c10afb4882dd61f54cf25a25a633ab316de8ce"}
Nov 01 05:20:10.159641 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:10.159591    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-p6p56" podStartSLOduration=2.159574016 podStartE2EDuration="2.159574016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:20:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:20:10.109972508 +0000 UTC m=+988.180769305" watchObservedRunningTime="2026-11-01 05:20:10.159574016 +0000 UTC m=+988.230370738"
Nov 01 05:20:10.555326 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:10.555290    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:20:10.658745 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:10.658712    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-p6p56"
Nov 01 05:20:10.659834 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:10.659815    5689 patch_prober.go:29] interesting pod/e2e-worker-w4-a48099-openshift-default-577864759-p6p56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Nov 01 05:20:10.659896 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:10.659867    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-p6p56" podUID="fc051a97-b093-4019-957f-63b017c153a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Nov 01 05:20:11.658632 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:11.658595    5689 patch_prober.go:29] interesting pod/e2e-worker-w4-a48099-openshift-default-577864759-p6p56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Nov 01 05:20:11.659028 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:11.658650    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-p6p56" podUID="fc051a97-b093-4019-957f-63b017c153a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Nov 01 05:20:12.658483 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:12.658430    5689 patch_prober.go:29] interesting pod/e2e-worker-w4-a48099-openshift-default-577864759-p6p56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Nov 01 05:20:12.658657 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:12.658513    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-p6p56" podUID="fc051a97-b093-4019-957f-63b017c153a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Nov 01 05:20:13.658707 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:13.658672    5689 patch_prober.go:29] interesting pod/e2e-worker-w4-a48099-openshift-default-577864759-p6p56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Nov 01 05:20:13.659128 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:13.658743    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-p6p56" podUID="fc051a97-b093-4019-957f-63b017c153a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Nov 01 05:20:13.731534 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:13.731491    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:20:13.911563 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:13.911446    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:20:14.111595 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:14.111559    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2864b6-d5585ff4c-x2hfw"]
Nov 01 05:20:14.115618 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:14.115597    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2864b6-d5585ff4c-x2hfw"
Nov 01 05:20:14.128958 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:14.128935    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adbbc4d7-e023-4063-be79-7da0aa65cd03-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2864b6-d5585ff4c-x2hfw\" (UID: \"adbbc4d7-e023-4063-be79-7da0aa65cd03\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2864b6-d5585ff4c-x2hfw"
Nov 01 05:20:14.129050 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:14.128971    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g982j\" (UniqueName: \"kubernetes.io/projected/adbbc4d7-e023-4063-be79-7da0aa65cd03-kube-api-access-g982j\") pod \"payload-pre-processing-e2e-ipp-a-2864b6-d5585ff4c-x2hfw\" (UID: \"adbbc4d7-e023-4063-be79-7da0aa65cd03\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2864b6-d5585ff4c-x2hfw"
Nov 01 05:20:14.167701 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:14.167652    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-2864b6\""
Nov 01 05:20:14.229669 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:14.229601    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-2864b6-dockercfg-nv59s\""
Nov 01 05:20:14.244544 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:14.234245    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2864b6-d5585ff4c-x2hfw"]
Nov 01 05:20:14.296509 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:14.281055    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adbbc4d7-e023-4063-be79-7da0aa65cd03-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2864b6-d5585ff4c-x2hfw\" (UID: \"adbbc4d7-e023-4063-be79-7da0aa65cd03\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2864b6-d5585ff4c-x2hfw"
Nov 01 05:20:14.296509 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:14.281122    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g982j\" (UniqueName: \"kubernetes.io/projected/adbbc4d7-e023-4063-be79-7da0aa65cd03-kube-api-access-g982j\") pod \"payload-pre-processing-e2e-ipp-a-2864b6-d5585ff4c-x2hfw\" (UID: \"adbbc4d7-e023-4063-be79-7da0aa65cd03\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2864b6-d5585ff4c-x2hfw"
Nov 01 05:20:14.296509 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:14.282412    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adbbc4d7-e023-4063-be79-7da0aa65cd03-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2864b6-d5585ff4c-x2hfw\" (UID: \"adbbc4d7-e023-4063-be79-7da0aa65cd03\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2864b6-d5585ff4c-x2hfw"
Nov 01 05:20:14.342049 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:14.342021    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g982j\" (UniqueName: \"kubernetes.io/projected/adbbc4d7-e023-4063-be79-7da0aa65cd03-kube-api-access-g982j\") pod \"payload-pre-processing-e2e-ipp-a-2864b6-d5585ff4c-x2hfw\" (UID: \"adbbc4d7-e023-4063-be79-7da0aa65cd03\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2864b6-d5585ff4c-x2hfw"
Nov 01 05:20:14.429277 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:14.429187    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2864b6-d5585ff4c-x2hfw"
Nov 01 05:20:14.658617 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:14.658587    5689 patch_prober.go:29] interesting pod/e2e-worker-w4-a48099-openshift-default-577864759-p6p56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Nov 01 05:20:14.658785 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:14.658638    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-p6p56" podUID="fc051a97-b093-4019-957f-63b017c153a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Nov 01 05:20:14.828434 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:14.828410    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2864b6-d5585ff4c-x2hfw"]
Nov 01 05:20:14.831285 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:20:14.831257    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadbbc4d7_e023_4063_be79_7da0aa65cd03.slice/crio-51e58bdfc26061adf91e674420698a00f0b270f8b8092a12267831e0c015e557 WatchSource:0}: Error finding container 51e58bdfc26061adf91e674420698a00f0b270f8b8092a12267831e0c015e557: Status 404 returned error can't find the container with id 51e58bdfc26061adf91e674420698a00f0b270f8b8092a12267831e0c015e557
Nov 01 05:20:14.960847 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:14.960817    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2864b6-d5585ff4c-x2hfw" event={"ID":"adbbc4d7-e023-4063-be79-7da0aa65cd03","Type":"ContainerStarted","Data":"dc03bd09ba0c0466c231814e97da0a917631ec77c183d4b1f40bf9312ca36f7a"}
Nov 01 05:20:14.960971 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:14.960875    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2864b6-d5585ff4c-x2hfw" event={"ID":"adbbc4d7-e023-4063-be79-7da0aa65cd03","Type":"ContainerStarted","Data":"51e58bdfc26061adf91e674420698a00f0b270f8b8092a12267831e0c015e557"}
Nov 01 05:20:14.960971 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:14.960919    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2864b6-d5585ff4c-x2hfw"
Nov 01 05:20:15.080072 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:15.079959    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2864b6-d5585ff4c-x2hfw" podStartSLOduration=1.079944512 podStartE2EDuration="1.079944512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:20:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:20:15.077751606 +0000 UTC m=+993.148548347" watchObservedRunningTime="2026-11-01 05:20:15.079944512 +0000 UTC m=+993.150741229"
Nov 01 05:20:15.420610 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:15.420580    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:20:15.664476 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:15.658846    5689 patch_prober.go:29] interesting pod/e2e-worker-w4-a48099-openshift-default-577864759-p6p56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Nov 01 05:20:15.664476 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:15.658905    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-p6p56" podUID="fc051a97-b093-4019-957f-63b017c153a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Nov 01 05:20:16.179205 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:16.179176    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:20:16.662909 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:16.662884    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-p6p56"
Nov 01 05:20:16.663177 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:16.663157    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-p6p56"
Nov 01 05:20:16.663907 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:16.663892    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-p6p56"
Nov 01 05:20:17.297204 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:17.297174    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4dbthwg"]
Nov 01 05:20:17.301164 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:17.301148    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4dbthwg"
Nov 01 05:20:17.353685 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:17.353662    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2fd4c9\"/\"routing-overlay\""
Nov 01 05:20:17.353685 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:17.353663    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2fd4c9\"/\"payload-processing-external-model-plugins-e2e-extmod-b-2fd4c9\""
Nov 01 05:20:17.353685 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:17.353690    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2fd4c9\"/\"kube-root-ca.crt\""
Nov 01 05:20:17.353889 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:17.353690    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2fd4c9\"/\"openshift-service-ca.crt\""
Nov 01 05:20:17.353889 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:17.353667    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2fd4c9\"/\"agc-cred-e326123de79d\""
Nov 01 05:20:17.353889 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:17.353880    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6d149758-c12f-4374-80ac-9da5bbea5071-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4dbthwg\" (UID: \"6d149758-c12f-4374-80ac-9da5bbea5071\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4dbthwg"
Nov 01 05:20:17.354019 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:17.354004    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6d149758-c12f-4374-80ac-9da5bbea5071-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4dbthwg\" (UID: \"6d149758-c12f-4374-80ac-9da5bbea5071\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4dbthwg"
Nov 01 05:20:17.354055 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:17.354044    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d149758-c12f-4374-80ac-9da5bbea5071-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4dbthwg\" (UID: \"6d149758-c12f-4374-80ac-9da5bbea5071\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4dbthwg"
Nov 01 05:20:17.354086 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:17.354070    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgsg6\" (UniqueName: \"kubernetes.io/projected/6d149758-c12f-4374-80ac-9da5bbea5071-kube-api-access-vgsg6\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4dbthwg\" (UID: \"6d149758-c12f-4374-80ac-9da5bbea5071\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4dbthwg"
Nov 01 05:20:17.356314 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:17.356295    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2fd4c9\"/\"payload-processing-external-model-e2e--928c4e32-dockercfg-r25vt\""
Nov 01 05:20:17.359022 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:17.359001    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4dbthwg"]
Nov 01 05:20:17.455832 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:17.455797    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6d149758-c12f-4374-80ac-9da5bbea5071-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4dbthwg\" (UID: \"6d149758-c12f-4374-80ac-9da5bbea5071\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4dbthwg"
Nov 01 05:20:17.455989 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:17.455908    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6d149758-c12f-4374-80ac-9da5bbea5071-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4dbthwg\" (UID: \"6d149758-c12f-4374-80ac-9da5bbea5071\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4dbthwg"
Nov 01 05:20:17.455989 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:17.455959    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d149758-c12f-4374-80ac-9da5bbea5071-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4dbthwg\" (UID: \"6d149758-c12f-4374-80ac-9da5bbea5071\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4dbthwg"
Nov 01 05:20:17.456108 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:17.455997    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgsg6\" (UniqueName: \"kubernetes.io/projected/6d149758-c12f-4374-80ac-9da5bbea5071-kube-api-access-vgsg6\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4dbthwg\" (UID: \"6d149758-c12f-4374-80ac-9da5bbea5071\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4dbthwg"
Nov 01 05:20:17.456735 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:17.456709    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6d149758-c12f-4374-80ac-9da5bbea5071-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4dbthwg\" (UID: \"6d149758-c12f-4374-80ac-9da5bbea5071\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4dbthwg"
Nov 01 05:20:17.456838 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:17.456805    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d149758-c12f-4374-80ac-9da5bbea5071-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4dbthwg\" (UID: \"6d149758-c12f-4374-80ac-9da5bbea5071\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4dbthwg"
Nov 01 05:20:17.458359 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:17.458337    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6d149758-c12f-4374-80ac-9da5bbea5071-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4dbthwg\" (UID: \"6d149758-c12f-4374-80ac-9da5bbea5071\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4dbthwg"
Nov 01 05:20:17.516687 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:17.516665    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgsg6\" (UniqueName: \"kubernetes.io/projected/6d149758-c12f-4374-80ac-9da5bbea5071-kube-api-access-vgsg6\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4dbthwg\" (UID: \"6d149758-c12f-4374-80ac-9da5bbea5071\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4dbthwg"
Nov 01 05:20:17.614791 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:17.614721    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4dbthwg"
Nov 01 05:20:17.989774 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:17.989746    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4dbthwg"]
Nov 01 05:20:17.992850 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:20:17.992815    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d149758_c12f_4374_80ac_9da5bbea5071.slice/crio-af7fc51da28853459bb5ca5664841d64959a3d8251a0163716fcc0596edc7174 WatchSource:0}: Error finding container af7fc51da28853459bb5ca5664841d64959a3d8251a0163716fcc0596edc7174: Status 404 returned error can't find the container with id af7fc51da28853459bb5ca5664841d64959a3d8251a0163716fcc0596edc7174
Nov 01 05:20:18.985115 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:18.985080    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4dbthwg" event={"ID":"6d149758-c12f-4374-80ac-9da5bbea5071","Type":"ContainerStarted","Data":"9e137522254d8a0c07d63dbd78dba610a35ccc63212e68520806625250572e7f"}
Nov 01 05:20:18.985511 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:18.985139    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4dbthwg"
Nov 01 05:20:18.985511 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:18.985159    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4dbthwg" event={"ID":"6d149758-c12f-4374-80ac-9da5bbea5071","Type":"ContainerStarted","Data":"af7fc51da28853459bb5ca5664841d64959a3d8251a0163716fcc0596edc7174"}
Nov 01 05:20:19.102689 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:19.102644    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4dbthwg" podStartSLOduration=2.102631467 podStartE2EDuration="2.102631467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:20:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:20:19.101066861 +0000 UTC m=+997.171863583" watchObservedRunningTime="2026-11-01 05:20:19.102631467 +0000 UTC m=+997.173428184"
Nov 01 05:20:20.226517 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:20.226286    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:20:25.056308 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:25.056270    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:20:25.969954 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:25.969925    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2864b6-d5585ff4c-x2hfw"
Nov 01 05:20:29.719860 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:29.719822    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:20:30.027866 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:30.027784    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4dbthwg"
Nov 01 05:20:30.068089 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:20:30.068054    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:20:32.528147 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:32.528113    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:20:34.512322 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:34.512282    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:20:39.273341 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:39.273308    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:20:44.140230 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:44.140197    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:20:45.434190 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:45.434158    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:20:48.718618 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:48.718585    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2864b6-6d657dbf4d-n6lvj"]
Nov 01 05:20:48.723659 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:48.723644    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2864b6-6d657dbf4d-n6lvj"
Nov 01 05:20:48.775773 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:48.775748    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-2864b6-serving-cert\""
Nov 01 05:20:48.827511 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:48.827485    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2864b6-6d657dbf4d-n6lvj"]
Nov 01 05:20:48.845022 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:48.844996    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/137ed9b9-0685-4ac6-a78e-e76348d77583-metrics-certs\") pod \"maas-api-e2e-ipp-a-2864b6-6d657dbf4d-n6lvj\" (UID: \"137ed9b9-0685-4ac6-a78e-e76348d77583\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2864b6-6d657dbf4d-n6lvj"
Nov 01 05:20:48.845190 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:48.845169    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/137ed9b9-0685-4ac6-a78e-e76348d77583-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-2864b6-6d657dbf4d-n6lvj\" (UID: \"137ed9b9-0685-4ac6-a78e-e76348d77583\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2864b6-6d657dbf4d-n6lvj"
Nov 01 05:20:48.845248 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:48.845203    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/137ed9b9-0685-4ac6-a78e-e76348d77583-maas-api-tls\") pod \"maas-api-e2e-ipp-a-2864b6-6d657dbf4d-n6lvj\" (UID: \"137ed9b9-0685-4ac6-a78e-e76348d77583\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2864b6-6d657dbf4d-n6lvj"
Nov 01 05:20:48.845248 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:48.845234    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr92r\" (UniqueName: \"kubernetes.io/projected/137ed9b9-0685-4ac6-a78e-e76348d77583-kube-api-access-fr92r\") pod \"maas-api-e2e-ipp-a-2864b6-6d657dbf4d-n6lvj\" (UID: \"137ed9b9-0685-4ac6-a78e-e76348d77583\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2864b6-6d657dbf4d-n6lvj"
Nov 01 05:20:48.939740 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:48.939707    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:20:48.945998 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:48.945972    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/137ed9b9-0685-4ac6-a78e-e76348d77583-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-2864b6-6d657dbf4d-n6lvj\" (UID: \"137ed9b9-0685-4ac6-a78e-e76348d77583\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2864b6-6d657dbf4d-n6lvj"
Nov 01 05:20:48.946144 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:48.946020    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/137ed9b9-0685-4ac6-a78e-e76348d77583-maas-api-tls\") pod \"maas-api-e2e-ipp-a-2864b6-6d657dbf4d-n6lvj\" (UID: \"137ed9b9-0685-4ac6-a78e-e76348d77583\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2864b6-6d657dbf4d-n6lvj"
Nov 01 05:20:48.946144 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:48.946074    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr92r\" (UniqueName: \"kubernetes.io/projected/137ed9b9-0685-4ac6-a78e-e76348d77583-kube-api-access-fr92r\") pod \"maas-api-e2e-ipp-a-2864b6-6d657dbf4d-n6lvj\" (UID: \"137ed9b9-0685-4ac6-a78e-e76348d77583\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2864b6-6d657dbf4d-n6lvj"
Nov 01 05:20:48.946266 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:20:48.946241    5689 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-2864b6-serving-cert: secret "maas-api-e2e-ipp-a-2864b6-serving-cert" not found
Nov 01 05:20:48.946323 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:48.946286    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/137ed9b9-0685-4ac6-a78e-e76348d77583-metrics-certs\") pod \"maas-api-e2e-ipp-a-2864b6-6d657dbf4d-n6lvj\" (UID: \"137ed9b9-0685-4ac6-a78e-e76348d77583\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2864b6-6d657dbf4d-n6lvj"
Nov 01 05:20:48.946323 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:20:48.946295    5689 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/137ed9b9-0685-4ac6-a78e-e76348d77583-maas-api-tls podName:137ed9b9-0685-4ac6-a78e-e76348d77583 nodeName:}" failed. No retries permitted until 2026-11-01 05:20:49.446277482 +0000 UTC m=+1027.517074195 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/137ed9b9-0685-4ac6-a78e-e76348d77583-maas-api-tls") pod "maas-api-e2e-ipp-a-2864b6-6d657dbf4d-n6lvj" (UID: "137ed9b9-0685-4ac6-a78e-e76348d77583") : secret "maas-api-e2e-ipp-a-2864b6-serving-cert" not found
Nov 01 05:20:48.946756 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:48.946732    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/137ed9b9-0685-4ac6-a78e-e76348d77583-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-2864b6-6d657dbf4d-n6lvj\" (UID: \"137ed9b9-0685-4ac6-a78e-e76348d77583\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2864b6-6d657dbf4d-n6lvj"
Nov 01 05:20:48.948866 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:48.948844    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/137ed9b9-0685-4ac6-a78e-e76348d77583-metrics-certs\") pod \"maas-api-e2e-ipp-a-2864b6-6d657dbf4d-n6lvj\" (UID: \"137ed9b9-0685-4ac6-a78e-e76348d77583\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2864b6-6d657dbf4d-n6lvj"
Nov 01 05:20:49.006509 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:49.006418    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr92r\" (UniqueName: \"kubernetes.io/projected/137ed9b9-0685-4ac6-a78e-e76348d77583-kube-api-access-fr92r\") pod \"maas-api-e2e-ipp-a-2864b6-6d657dbf4d-n6lvj\" (UID: \"137ed9b9-0685-4ac6-a78e-e76348d77583\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2864b6-6d657dbf4d-n6lvj"
Nov 01 05:20:49.120618 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:49.120585    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:20:49.297359 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:49.297278    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:20:49.452659 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:49.452631    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/137ed9b9-0685-4ac6-a78e-e76348d77583-maas-api-tls\") pod \"maas-api-e2e-ipp-a-2864b6-6d657dbf4d-n6lvj\" (UID: \"137ed9b9-0685-4ac6-a78e-e76348d77583\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2864b6-6d657dbf4d-n6lvj"
Nov 01 05:20:49.455142 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:49.455110    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/137ed9b9-0685-4ac6-a78e-e76348d77583-maas-api-tls\") pod \"maas-api-e2e-ipp-a-2864b6-6d657dbf4d-n6lvj\" (UID: \"137ed9b9-0685-4ac6-a78e-e76348d77583\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2864b6-6d657dbf4d-n6lvj"
Nov 01 05:20:49.636912 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:49.636841    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2864b6-6d657dbf4d-n6lvj"
Nov 01 05:20:50.016588 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:50.016562    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2864b6-6d657dbf4d-n6lvj"]
Nov 01 05:20:50.019115 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:20:50.019086    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod137ed9b9_0685_4ac6_a78e_e76348d77583.slice/crio-25f6214f5257fae514c3f248c067cddb2a1cd0223bc1679785635c5f6a7c58bd WatchSource:0}: Error finding container 25f6214f5257fae514c3f248c067cddb2a1cd0223bc1679785635c5f6a7c58bd: Status 404 returned error can't find the container with id 25f6214f5257fae514c3f248c067cddb2a1cd0223bc1679785635c5f6a7c58bd
Nov 01 05:20:50.020513 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:50.020495    5689 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 05:20:50.231252 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:50.231215    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2864b6-6d657dbf4d-n6lvj" event={"ID":"137ed9b9-0685-4ac6-a78e-e76348d77583","Type":"ContainerStarted","Data":"25f6214f5257fae514c3f248c067cddb2a1cd0223bc1679785635c5f6a7c58bd"}
Nov 01 05:20:51.238341 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:51.238259    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2864b6-6d657dbf4d-n6lvj" event={"ID":"137ed9b9-0685-4ac6-a78e-e76348d77583","Type":"ContainerStarted","Data":"4e3f021fc9263beb34dc693d444e61487df36d1d0497752f23fba396ef368eab"}
Nov 01 05:20:51.238813 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:51.238387    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2864b6-6d657dbf4d-n6lvj"
Nov 01 05:20:51.363371 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:51.363300    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2864b6-6d657dbf4d-n6lvj" podStartSLOduration=2.440376835 podStartE2EDuration="3.363282308s" totalImagesPullingTime="922.905473ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:20:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:20:51.362839103 +0000 UTC m=+1029.433635824" watchObservedRunningTime="2026-11-01 05:20:51.363282308 +0000 UTC m=+1029.434079031"
Nov 01 05:20:53.996819 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:53.996780    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:20:56.701656 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:56.701620    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:20:56.880848 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:56.880816    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:20:57.251721 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:57.251696    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2864b6-6d657dbf4d-n6lvj"
Nov 01 05:20:58.836537 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:58.831595    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:20:59.000889 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:59.000849    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f24gh"]
Nov 01 05:20:59.001138 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:59.001114    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f24gh" podUID="7758bc0a-0b7f-45d6-b688-53936075c3c7" containerName="istio-proxy" containerID="cri-o://05ba11e9854e456be39666e580103631670cdec4b09a850055b05a8a5f21d0ee" gracePeriod=30
Nov 01 05:20:59.050622 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:59.050588    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jxzw"]
Nov 01 05:20:59.050803 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:59.050781    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jxzw" podUID="54667545-c5d3-4893-9ade-e16c82cb5b93" containerName="istio-proxy" containerID="cri-o://ff5e79c1f7e800789000628267819a6eceeee654a883f86558a39889f235a2e1" gracePeriod=30
Nov 01 05:20:59.052589 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:59.052571    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-2jwpv"]
Nov 01 05:20:59.052763 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:59.052744    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-2jwpv" podUID="5944a5e4-357d-4c82-930f-17b1f022f376" containerName="istio-proxy" containerID="cri-o://472eba9828ecd5b8ab3b41248e35904b9176350ceab42b5d8ccc6207f5f2c50c" gracePeriod=30
Nov 01 05:20:59.099591 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:59.099520    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5t46"]
Nov 01 05:20:59.099769 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:59.099672    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5t46" podUID="231ce25c-5629-4780-9607-e5cd270afba5" containerName="istio-proxy" containerID="cri-o://96c9d077aea787fd04dfd73dbba04c445ea921469b1f822a5db75386642ee5a0" gracePeriod=30
Nov 01 05:20:59.278710 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:20:59.278639    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:21:00.069100 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:21:00.069068    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:21:03.647580 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:03.647480    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:21:04.314110 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.314013    5689 generic.go:356] "Generic (PLEG): container finished" podID="7758bc0a-0b7f-45d6-b688-53936075c3c7" containerID="05ba11e9854e456be39666e580103631670cdec4b09a850055b05a8a5f21d0ee" exitCode=0
Nov 01 05:21:04.314110 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.314087    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f24gh" event={"ID":"7758bc0a-0b7f-45d6-b688-53936075c3c7","Type":"ContainerDied","Data":"05ba11e9854e456be39666e580103631670cdec4b09a850055b05a8a5f21d0ee"}
Nov 01 05:21:04.316387 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.316353    5689 generic.go:356] "Generic (PLEG): container finished" podID="5944a5e4-357d-4c82-930f-17b1f022f376" containerID="472eba9828ecd5b8ab3b41248e35904b9176350ceab42b5d8ccc6207f5f2c50c" exitCode=0
Nov 01 05:21:04.316542 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.316427    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-2jwpv" event={"ID":"5944a5e4-357d-4c82-930f-17b1f022f376","Type":"ContainerDied","Data":"472eba9828ecd5b8ab3b41248e35904b9176350ceab42b5d8ccc6207f5f2c50c"}
Nov 01 05:21:04.318331 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.318309    5689 generic.go:356] "Generic (PLEG): container finished" podID="54667545-c5d3-4893-9ade-e16c82cb5b93" containerID="ff5e79c1f7e800789000628267819a6eceeee654a883f86558a39889f235a2e1" exitCode=0
Nov 01 05:21:04.318507 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.318386    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jxzw" event={"ID":"54667545-c5d3-4893-9ade-e16c82cb5b93","Type":"ContainerDied","Data":"ff5e79c1f7e800789000628267819a6eceeee654a883f86558a39889f235a2e1"}
Nov 01 05:21:04.320436 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.320415    5689 generic.go:356] "Generic (PLEG): container finished" podID="231ce25c-5629-4780-9607-e5cd270afba5" containerID="96c9d077aea787fd04dfd73dbba04c445ea921469b1f822a5db75386642ee5a0" exitCode=0
Nov 01 05:21:04.320571 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.320546    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5t46" event={"ID":"231ce25c-5629-4780-9607-e5cd270afba5","Type":"ContainerDied","Data":"96c9d077aea787fd04dfd73dbba04c445ea921469b1f822a5db75386642ee5a0"}
Nov 01 05:21:04.353259 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.353238    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f24gh"
Nov 01 05:21:04.384943 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.384916    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-2jwpv"
Nov 01 05:21:04.429041 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.429016    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jxzw"
Nov 01 05:21:04.437391 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.437368    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5t46"
Nov 01 05:21:04.446603 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.446572    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7758bc0a-0b7f-45d6-b688-53936075c3c7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7758bc0a-0b7f-45d6-b688-53936075c3c7-credential-socket\") pod \"7758bc0a-0b7f-45d6-b688-53936075c3c7\" (UID: \"7758bc0a-0b7f-45d6-b688-53936075c3c7\") "
Nov 01 05:21:04.446729 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.446693    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7758bc0a-0b7f-45d6-b688-53936075c3c7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7758bc0a-0b7f-45d6-b688-53936075c3c7-workload-socket\") pod \"7758bc0a-0b7f-45d6-b688-53936075c3c7\" (UID: \"7758bc0a-0b7f-45d6-b688-53936075c3c7\") "
Nov 01 05:21:04.446809 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.446726    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7758bc0a-0b7f-45d6-b688-53936075c3c7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7758bc0a-0b7f-45d6-b688-53936075c3c7-istiod-ca-cert\") pod \"7758bc0a-0b7f-45d6-b688-53936075c3c7\" (UID: \"7758bc0a-0b7f-45d6-b688-53936075c3c7\") "
Nov 01 05:21:04.446809 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.446758    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7758bc0a-0b7f-45d6-b688-53936075c3c7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7758bc0a-0b7f-45d6-b688-53936075c3c7-istio-data\") pod \"7758bc0a-0b7f-45d6-b688-53936075c3c7\" (UID: \"7758bc0a-0b7f-45d6-b688-53936075c3c7\") "
Nov 01 05:21:04.446925 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.446814    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7758bc0a-0b7f-45d6-b688-53936075c3c7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7758bc0a-0b7f-45d6-b688-53936075c3c7-istio-envoy\") pod \"7758bc0a-0b7f-45d6-b688-53936075c3c7\" (UID: \"7758bc0a-0b7f-45d6-b688-53936075c3c7\") "
Nov 01 05:21:04.446925 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.446842    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7758bc0a-0b7f-45d6-b688-53936075c3c7-credential-socket" pod "7758bc0a-0b7f-45d6-b688-53936075c3c7" (UID: "7758bc0a-0b7f-45d6-b688-53936075c3c7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:21:04.447037 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.446960    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7758bc0a-0b7f-45d6-b688-53936075c3c7-istio-token\" (UniqueName: \"kubernetes.io/projected/7758bc0a-0b7f-45d6-b688-53936075c3c7-istio-token\") pod \"7758bc0a-0b7f-45d6-b688-53936075c3c7\" (UID: \"7758bc0a-0b7f-45d6-b688-53936075c3c7\") "
Nov 01 05:21:04.447037 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.446998    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7758bc0a-0b7f-45d6-b688-53936075c3c7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7758bc0a-0b7f-45d6-b688-53936075c3c7-workload-certs\") pod \"7758bc0a-0b7f-45d6-b688-53936075c3c7\" (UID: \"7758bc0a-0b7f-45d6-b688-53936075c3c7\") "
Nov 01 05:21:04.447037 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.447026    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7758bc0a-0b7f-45d6-b688-53936075c3c7-kube-api-access-qfmbf\" (UniqueName: \"kubernetes.io/projected/7758bc0a-0b7f-45d6-b688-53936075c3c7-kube-api-access-qfmbf\") pod \"7758bc0a-0b7f-45d6-b688-53936075c3c7\" (UID: \"7758bc0a-0b7f-45d6-b688-53936075c3c7\") "
Nov 01 05:21:04.447190 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.447068    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7758bc0a-0b7f-45d6-b688-53936075c3c7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7758bc0a-0b7f-45d6-b688-53936075c3c7-istio-podinfo\") pod \"7758bc0a-0b7f-45d6-b688-53936075c3c7\" (UID: \"7758bc0a-0b7f-45d6-b688-53936075c3c7\") "
Nov 01 05:21:04.447190 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.447092    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7758bc0a-0b7f-45d6-b688-53936075c3c7-istio-data" pod "7758bc0a-0b7f-45d6-b688-53936075c3c7" (UID: "7758bc0a-0b7f-45d6-b688-53936075c3c7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:21:04.447190 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.447179    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7758bc0a-0b7f-45d6-b688-53936075c3c7-istiod-ca-cert" pod "7758bc0a-0b7f-45d6-b688-53936075c3c7" (UID: "7758bc0a-0b7f-45d6-b688-53936075c3c7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:21:04.447518 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.447494    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7758bc0a-0b7f-45d6-b688-53936075c3c7-workload-socket" pod "7758bc0a-0b7f-45d6-b688-53936075c3c7" (UID: "7758bc0a-0b7f-45d6-b688-53936075c3c7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:21:04.447733 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.447710    5689 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7758bc0a-0b7f-45d6-b688-53936075c3c7-credential-socket\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:21:04.447812 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.447740    5689 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7758bc0a-0b7f-45d6-b688-53936075c3c7-workload-socket\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:21:04.447812 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.447754    5689 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7758bc0a-0b7f-45d6-b688-53936075c3c7-istiod-ca-cert\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:21:04.447812 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.447768    5689 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7758bc0a-0b7f-45d6-b688-53936075c3c7-istio-data\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:21:04.449330 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.449299    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7758bc0a-0b7f-45d6-b688-53936075c3c7-workload-certs" pod "7758bc0a-0b7f-45d6-b688-53936075c3c7" (UID: "7758bc0a-0b7f-45d6-b688-53936075c3c7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:21:04.449878 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.449855    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7758bc0a-0b7f-45d6-b688-53936075c3c7-kube-api-access-qfmbf" pod "7758bc0a-0b7f-45d6-b688-53936075c3c7" (UID: "7758bc0a-0b7f-45d6-b688-53936075c3c7"). InnerVolumeSpecName "kube-api-access-qfmbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:21:04.450258 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.450228    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7758bc0a-0b7f-45d6-b688-53936075c3c7-istio-token" pod "7758bc0a-0b7f-45d6-b688-53936075c3c7" (UID: "7758bc0a-0b7f-45d6-b688-53936075c3c7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:21:04.450415 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.450394    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7758bc0a-0b7f-45d6-b688-53936075c3c7-istio-podinfo" pod "7758bc0a-0b7f-45d6-b688-53936075c3c7" (UID: "7758bc0a-0b7f-45d6-b688-53936075c3c7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 05:21:04.451555 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.451536    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7758bc0a-0b7f-45d6-b688-53936075c3c7-istio-envoy" pod "7758bc0a-0b7f-45d6-b688-53936075c3c7" (UID: "7758bc0a-0b7f-45d6-b688-53936075c3c7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:21:04.549131 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.549054    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/231ce25c-5629-4780-9607-e5cd270afba5-istio-token\" (UniqueName: \"kubernetes.io/projected/231ce25c-5629-4780-9607-e5cd270afba5-istio-token\") pod \"231ce25c-5629-4780-9607-e5cd270afba5\" (UID: \"231ce25c-5629-4780-9607-e5cd270afba5\") "
Nov 01 05:21:04.549131 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.549094    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5944a5e4-357d-4c82-930f-17b1f022f376-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5944a5e4-357d-4c82-930f-17b1f022f376-workload-socket\") pod \"5944a5e4-357d-4c82-930f-17b1f022f376\" (UID: \"5944a5e4-357d-4c82-930f-17b1f022f376\") "
Nov 01 05:21:04.549131 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.549115    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54667545-c5d3-4893-9ade-e16c82cb5b93-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54667545-c5d3-4893-9ade-e16c82cb5b93-istio-data\") pod \"54667545-c5d3-4893-9ade-e16c82cb5b93\" (UID: \"54667545-c5d3-4893-9ade-e16c82cb5b93\") "
Nov 01 05:21:04.549400 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.549146    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54667545-c5d3-4893-9ade-e16c82cb5b93-kube-api-access-fdkdm\" (UniqueName: \"kubernetes.io/projected/54667545-c5d3-4893-9ade-e16c82cb5b93-kube-api-access-fdkdm\") pod \"54667545-c5d3-4893-9ade-e16c82cb5b93\" (UID: \"54667545-c5d3-4893-9ade-e16c82cb5b93\") "
Nov 01 05:21:04.549400 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.549172    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5944a5e4-357d-4c82-930f-17b1f022f376-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5944a5e4-357d-4c82-930f-17b1f022f376-istio-data\") pod \"5944a5e4-357d-4c82-930f-17b1f022f376\" (UID: \"5944a5e4-357d-4c82-930f-17b1f022f376\") "
Nov 01 05:21:04.549400 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.549205    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/54667545-c5d3-4893-9ade-e16c82cb5b93-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54667545-c5d3-4893-9ade-e16c82cb5b93-istio-podinfo\") pod \"54667545-c5d3-4893-9ade-e16c82cb5b93\" (UID: \"54667545-c5d3-4893-9ade-e16c82cb5b93\") "
Nov 01 05:21:04.549400 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.549252    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54667545-c5d3-4893-9ade-e16c82cb5b93-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54667545-c5d3-4893-9ade-e16c82cb5b93-istio-envoy\") pod \"54667545-c5d3-4893-9ade-e16c82cb5b93\" (UID: \"54667545-c5d3-4893-9ade-e16c82cb5b93\") "
Nov 01 05:21:04.549400 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.549287    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5944a5e4-357d-4c82-930f-17b1f022f376-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5944a5e4-357d-4c82-930f-17b1f022f376-workload-certs\") pod \"5944a5e4-357d-4c82-930f-17b1f022f376\" (UID: \"5944a5e4-357d-4c82-930f-17b1f022f376\") "
Nov 01 05:21:04.549400 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.549328    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54667545-c5d3-4893-9ade-e16c82cb5b93-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54667545-c5d3-4893-9ade-e16c82cb5b93-workload-certs\") pod \"54667545-c5d3-4893-9ade-e16c82cb5b93\" (UID: \"54667545-c5d3-4893-9ade-e16c82cb5b93\") "
Nov 01 05:21:04.549400 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.549383    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5944a5e4-357d-4c82-930f-17b1f022f376-istio-token\" (UniqueName: \"kubernetes.io/projected/5944a5e4-357d-4c82-930f-17b1f022f376-istio-token\") pod \"5944a5e4-357d-4c82-930f-17b1f022f376\" (UID: \"5944a5e4-357d-4c82-930f-17b1f022f376\") "
Nov 01 05:21:04.549819 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.549392    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5944a5e4-357d-4c82-930f-17b1f022f376-workload-socket" pod "5944a5e4-357d-4c82-930f-17b1f022f376" (UID: "5944a5e4-357d-4c82-930f-17b1f022f376"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:21:04.549819 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.549410    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5944a5e4-357d-4c82-930f-17b1f022f376-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5944a5e4-357d-4c82-930f-17b1f022f376-istio-envoy\") pod \"5944a5e4-357d-4c82-930f-17b1f022f376\" (UID: \"5944a5e4-357d-4c82-930f-17b1f022f376\") "
Nov 01 05:21:04.549819 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.549445    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5944a5e4-357d-4c82-930f-17b1f022f376-kube-api-access-q9dtz\" (UniqueName: \"kubernetes.io/projected/5944a5e4-357d-4c82-930f-17b1f022f376-kube-api-access-q9dtz\") pod \"5944a5e4-357d-4c82-930f-17b1f022f376\" (UID: \"5944a5e4-357d-4c82-930f-17b1f022f376\") "
Nov 01 05:21:04.549819 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.549502    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/231ce25c-5629-4780-9607-e5cd270afba5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/231ce25c-5629-4780-9607-e5cd270afba5-istio-podinfo\") pod \"231ce25c-5629-4780-9607-e5cd270afba5\" (UID: \"231ce25c-5629-4780-9607-e5cd270afba5\") "
Nov 01 05:21:04.549819 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.549569    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/231ce25c-5629-4780-9607-e5cd270afba5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/231ce25c-5629-4780-9607-e5cd270afba5-credential-socket\") pod \"231ce25c-5629-4780-9607-e5cd270afba5\" (UID: \"231ce25c-5629-4780-9607-e5cd270afba5\") "
Nov 01 05:21:04.549819 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.549613    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5944a5e4-357d-4c82-930f-17b1f022f376-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5944a5e4-357d-4c82-930f-17b1f022f376-istiod-ca-cert\") pod \"5944a5e4-357d-4c82-930f-17b1f022f376\" (UID: \"5944a5e4-357d-4c82-930f-17b1f022f376\") "
Nov 01 05:21:04.549819 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.549662    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54667545-c5d3-4893-9ade-e16c82cb5b93-istio-token\" (UniqueName: \"kubernetes.io/projected/54667545-c5d3-4893-9ade-e16c82cb5b93-istio-token\") pod \"54667545-c5d3-4893-9ade-e16c82cb5b93\" (UID: \"54667545-c5d3-4893-9ade-e16c82cb5b93\") "
Nov 01 05:21:04.549819 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.549667    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5944a5e4-357d-4c82-930f-17b1f022f376-workload-certs" pod "5944a5e4-357d-4c82-930f-17b1f022f376" (UID: "5944a5e4-357d-4c82-930f-17b1f022f376"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:21:04.549819 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.549705    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/231ce25c-5629-4780-9607-e5cd270afba5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/231ce25c-5629-4780-9607-e5cd270afba5-istio-data\") pod \"231ce25c-5629-4780-9607-e5cd270afba5\" (UID: \"231ce25c-5629-4780-9607-e5cd270afba5\") "
Nov 01 05:21:04.549819 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.549734    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/231ce25c-5629-4780-9607-e5cd270afba5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/231ce25c-5629-4780-9607-e5cd270afba5-istiod-ca-cert\") pod \"231ce25c-5629-4780-9607-e5cd270afba5\" (UID: \"231ce25c-5629-4780-9607-e5cd270afba5\") "
Nov 01 05:21:04.549819 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.549764    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5944a5e4-357d-4c82-930f-17b1f022f376-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5944a5e4-357d-4c82-930f-17b1f022f376-istio-podinfo\") pod \"5944a5e4-357d-4c82-930f-17b1f022f376\" (UID: \"5944a5e4-357d-4c82-930f-17b1f022f376\") "
Nov 01 05:21:04.549819 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.549805    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5944a5e4-357d-4c82-930f-17b1f022f376-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5944a5e4-357d-4c82-930f-17b1f022f376-credential-socket\") pod \"5944a5e4-357d-4c82-930f-17b1f022f376\" (UID: \"5944a5e4-357d-4c82-930f-17b1f022f376\") "
Nov 01 05:21:04.550405 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.549834    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/231ce25c-5629-4780-9607-e5cd270afba5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/231ce25c-5629-4780-9607-e5cd270afba5-istio-envoy\") pod \"231ce25c-5629-4780-9607-e5cd270afba5\" (UID: \"231ce25c-5629-4780-9607-e5cd270afba5\") "
Nov 01 05:21:04.550405 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.549900    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54667545-c5d3-4893-9ade-e16c82cb5b93-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54667545-c5d3-4893-9ade-e16c82cb5b93-credential-socket\") pod \"54667545-c5d3-4893-9ade-e16c82cb5b93\" (UID: \"54667545-c5d3-4893-9ade-e16c82cb5b93\") "
Nov 01 05:21:04.550405 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.549929    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/231ce25c-5629-4780-9607-e5cd270afba5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/231ce25c-5629-4780-9607-e5cd270afba5-workload-socket\") pod \"231ce25c-5629-4780-9607-e5cd270afba5\" (UID: \"231ce25c-5629-4780-9607-e5cd270afba5\") "
Nov 01 05:21:04.550405 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.549960    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/231ce25c-5629-4780-9607-e5cd270afba5-kube-api-access-zvv4b\" (UniqueName: \"kubernetes.io/projected/231ce25c-5629-4780-9607-e5cd270afba5-kube-api-access-zvv4b\") pod \"231ce25c-5629-4780-9607-e5cd270afba5\" (UID: \"231ce25c-5629-4780-9607-e5cd270afba5\") "
Nov 01 05:21:04.550405 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.550027    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/231ce25c-5629-4780-9607-e5cd270afba5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/231ce25c-5629-4780-9607-e5cd270afba5-workload-certs\") pod \"231ce25c-5629-4780-9607-e5cd270afba5\" (UID: \"231ce25c-5629-4780-9607-e5cd270afba5\") "
Nov 01 05:21:04.550405 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.550084    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54667545-c5d3-4893-9ade-e16c82cb5b93-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54667545-c5d3-4893-9ade-e16c82cb5b93-istiod-ca-cert\") pod \"54667545-c5d3-4893-9ade-e16c82cb5b93\" (UID: \"54667545-c5d3-4893-9ade-e16c82cb5b93\") "
Nov 01 05:21:04.550405 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.550082    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5944a5e4-357d-4c82-930f-17b1f022f376-istio-data" pod "5944a5e4-357d-4c82-930f-17b1f022f376" (UID: "5944a5e4-357d-4c82-930f-17b1f022f376"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:21:04.550405 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.550161    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54667545-c5d3-4893-9ade-e16c82cb5b93-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54667545-c5d3-4893-9ade-e16c82cb5b93-workload-socket\") pod \"54667545-c5d3-4893-9ade-e16c82cb5b93\" (UID: \"54667545-c5d3-4893-9ade-e16c82cb5b93\") "
Nov 01 05:21:04.550405 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.550367    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/231ce25c-5629-4780-9607-e5cd270afba5-credential-socket" pod "231ce25c-5629-4780-9607-e5cd270afba5" (UID: "231ce25c-5629-4780-9607-e5cd270afba5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:21:04.550881 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.550537    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54667545-c5d3-4893-9ade-e16c82cb5b93-workload-certs" pod "54667545-c5d3-4893-9ade-e16c82cb5b93" (UID: "54667545-c5d3-4893-9ade-e16c82cb5b93"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:21:04.550881 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.550579    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54667545-c5d3-4893-9ade-e16c82cb5b93-workload-socket" pod "54667545-c5d3-4893-9ade-e16c82cb5b93" (UID: "54667545-c5d3-4893-9ade-e16c82cb5b93"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:21:04.550881 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.550583    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5944a5e4-357d-4c82-930f-17b1f022f376-istiod-ca-cert" pod "5944a5e4-357d-4c82-930f-17b1f022f376" (UID: "5944a5e4-357d-4c82-930f-17b1f022f376"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:21:04.550881 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.550630    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54667545-c5d3-4893-9ade-e16c82cb5b93-istio-data" pod "54667545-c5d3-4893-9ade-e16c82cb5b93" (UID: "54667545-c5d3-4893-9ade-e16c82cb5b93"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:21:04.550881 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.550826    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54667545-c5d3-4893-9ade-e16c82cb5b93-credential-socket" pod "54667545-c5d3-4893-9ade-e16c82cb5b93" (UID: "54667545-c5d3-4893-9ade-e16c82cb5b93"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:21:04.550881 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.550830    5689 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5944a5e4-357d-4c82-930f-17b1f022f376-istiod-ca-cert\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:21:04.550881 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.550864    5689 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7758bc0a-0b7f-45d6-b688-53936075c3c7-istio-envoy\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:21:04.550881 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.550880    5689 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54667545-c5d3-4893-9ade-e16c82cb5b93-workload-socket\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:21:04.551287 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.550895    5689 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7758bc0a-0b7f-45d6-b688-53936075c3c7-istio-token\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:21:04.551287 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.550909    5689 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7758bc0a-0b7f-45d6-b688-53936075c3c7-workload-certs\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:21:04.551287 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.550922    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfmbf\" (UniqueName: \"kubernetes.io/projected/7758bc0a-0b7f-45d6-b688-53936075c3c7-kube-api-access-qfmbf\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:21:04.551287 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.550937    5689 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5944a5e4-357d-4c82-930f-17b1f022f376-workload-socket\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:21:04.551287 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.550949    5689 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7758bc0a-0b7f-45d6-b688-53936075c3c7-istio-podinfo\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:21:04.551287 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.550962    5689 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54667545-c5d3-4893-9ade-e16c82cb5b93-istio-data\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:21:04.551287 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.550975    5689 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5944a5e4-357d-4c82-930f-17b1f022f376-istio-data\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:21:04.551287 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.550991    5689 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5944a5e4-357d-4c82-930f-17b1f022f376-workload-certs\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:21:04.551287 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.551004    5689 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54667545-c5d3-4893-9ade-e16c82cb5b93-workload-certs\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:21:04.551287 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.551017    5689 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/231ce25c-5629-4780-9607-e5cd270afba5-credential-socket\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:21:04.551799 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.551386    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/231ce25c-5629-4780-9607-e5cd270afba5-workload-socket" pod "231ce25c-5629-4780-9607-e5cd270afba5" (UID: "231ce25c-5629-4780-9607-e5cd270afba5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:21:04.552188 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.552162    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/231ce25c-5629-4780-9607-e5cd270afba5-workload-certs" pod "231ce25c-5629-4780-9607-e5cd270afba5" (UID: "231ce25c-5629-4780-9607-e5cd270afba5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:21:04.552607 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.552446    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54667545-c5d3-4893-9ade-e16c82cb5b93-istiod-ca-cert" pod "54667545-c5d3-4893-9ade-e16c82cb5b93" (UID: "54667545-c5d3-4893-9ade-e16c82cb5b93"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:21:04.553664 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.553369    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/231ce25c-5629-4780-9607-e5cd270afba5-istiod-ca-cert" pod "231ce25c-5629-4780-9607-e5cd270afba5" (UID: "231ce25c-5629-4780-9607-e5cd270afba5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:21:04.553922 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.553879    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/231ce25c-5629-4780-9607-e5cd270afba5-istio-data" pod "231ce25c-5629-4780-9607-e5cd270afba5" (UID: "231ce25c-5629-4780-9607-e5cd270afba5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:21:04.554328 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.554296    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5944a5e4-357d-4c82-930f-17b1f022f376-credential-socket" pod "5944a5e4-357d-4c82-930f-17b1f022f376" (UID: "5944a5e4-357d-4c82-930f-17b1f022f376"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:21:04.554417 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.554364    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/231ce25c-5629-4780-9607-e5cd270afba5-kube-api-access-zvv4b" pod "231ce25c-5629-4780-9607-e5cd270afba5" (UID: "231ce25c-5629-4780-9607-e5cd270afba5"). InnerVolumeSpecName "kube-api-access-zvv4b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:21:04.554993 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.554961    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/231ce25c-5629-4780-9607-e5cd270afba5-istio-token" pod "231ce25c-5629-4780-9607-e5cd270afba5" (UID: "231ce25c-5629-4780-9607-e5cd270afba5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:21:04.556670 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.556644    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54667545-c5d3-4893-9ade-e16c82cb5b93-istio-token" pod "54667545-c5d3-4893-9ade-e16c82cb5b93" (UID: "54667545-c5d3-4893-9ade-e16c82cb5b93"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:21:04.556779 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.556756    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54667545-c5d3-4893-9ade-e16c82cb5b93-istio-envoy" pod "54667545-c5d3-4893-9ade-e16c82cb5b93" (UID: "54667545-c5d3-4893-9ade-e16c82cb5b93"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:21:04.556779 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.556758    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5944a5e4-357d-4c82-930f-17b1f022f376-istio-token" pod "5944a5e4-357d-4c82-930f-17b1f022f376" (UID: "5944a5e4-357d-4c82-930f-17b1f022f376"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:21:04.556907 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.556871    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/231ce25c-5629-4780-9607-e5cd270afba5-istio-podinfo" pod "231ce25c-5629-4780-9607-e5cd270afba5" (UID: "231ce25c-5629-4780-9607-e5cd270afba5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 05:21:04.556957 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.556907    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54667545-c5d3-4893-9ade-e16c82cb5b93-kube-api-access-fdkdm" pod "54667545-c5d3-4893-9ade-e16c82cb5b93" (UID: "54667545-c5d3-4893-9ade-e16c82cb5b93"). InnerVolumeSpecName "kube-api-access-fdkdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:21:04.556957 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.556912    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5944a5e4-357d-4c82-930f-17b1f022f376-kube-api-access-q9dtz" pod "5944a5e4-357d-4c82-930f-17b1f022f376" (UID: "5944a5e4-357d-4c82-930f-17b1f022f376"). InnerVolumeSpecName "kube-api-access-q9dtz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:21:04.556957 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.556927    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/54667545-c5d3-4893-9ade-e16c82cb5b93-istio-podinfo" pod "54667545-c5d3-4893-9ade-e16c82cb5b93" (UID: "54667545-c5d3-4893-9ade-e16c82cb5b93"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 05:21:04.557089 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.557072    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5944a5e4-357d-4c82-930f-17b1f022f376-istio-podinfo" pod "5944a5e4-357d-4c82-930f-17b1f022f376" (UID: "5944a5e4-357d-4c82-930f-17b1f022f376"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 05:21:04.557313 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.557298    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/231ce25c-5629-4780-9607-e5cd270afba5-istio-envoy" pod "231ce25c-5629-4780-9607-e5cd270afba5" (UID: "231ce25c-5629-4780-9607-e5cd270afba5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:21:04.558626 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.558602    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5944a5e4-357d-4c82-930f-17b1f022f376-istio-envoy" pod "5944a5e4-357d-4c82-930f-17b1f022f376" (UID: "5944a5e4-357d-4c82-930f-17b1f022f376"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:21:04.652237 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.652201    5689 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/231ce25c-5629-4780-9607-e5cd270afba5-istio-token\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:21:04.652237 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.652232    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdkdm\" (UniqueName: \"kubernetes.io/projected/54667545-c5d3-4893-9ade-e16c82cb5b93-kube-api-access-fdkdm\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:21:04.652758 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.652247    5689 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54667545-c5d3-4893-9ade-e16c82cb5b93-istio-podinfo\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:21:04.652758 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.652259    5689 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54667545-c5d3-4893-9ade-e16c82cb5b93-istio-envoy\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:21:04.652758 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.652273    5689 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5944a5e4-357d-4c82-930f-17b1f022f376-istio-token\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:21:04.652758 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.652285    5689 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5944a5e4-357d-4c82-930f-17b1f022f376-istio-envoy\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:21:04.652758 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.652297    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9dtz\" (UniqueName: \"kubernetes.io/projected/5944a5e4-357d-4c82-930f-17b1f022f376-kube-api-access-q9dtz\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:21:04.652758 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.652310    5689 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/231ce25c-5629-4780-9607-e5cd270afba5-istio-podinfo\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:21:04.652758 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.652322    5689 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54667545-c5d3-4893-9ade-e16c82cb5b93-istio-token\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:21:04.652758 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.652337    5689 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/231ce25c-5629-4780-9607-e5cd270afba5-istio-data\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:21:04.652758 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.652349    5689 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/231ce25c-5629-4780-9607-e5cd270afba5-istiod-ca-cert\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:21:04.652758 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.652363    5689 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5944a5e4-357d-4c82-930f-17b1f022f376-istio-podinfo\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:21:04.652758 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.652378    5689 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5944a5e4-357d-4c82-930f-17b1f022f376-credential-socket\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:21:04.652758 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.652392    5689 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/231ce25c-5629-4780-9607-e5cd270afba5-istio-envoy\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:21:04.652758 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.652406    5689 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54667545-c5d3-4893-9ade-e16c82cb5b93-credential-socket\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:21:04.652758 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.652420    5689 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/231ce25c-5629-4780-9607-e5cd270afba5-workload-socket\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:21:04.652758 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.652432    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvv4b\" (UniqueName: \"kubernetes.io/projected/231ce25c-5629-4780-9607-e5cd270afba5-kube-api-access-zvv4b\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:21:04.652758 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.652446    5689 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/231ce25c-5629-4780-9607-e5cd270afba5-workload-certs\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:21:04.652758 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:04.652478    5689 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54667545-c5d3-4893-9ade-e16c82cb5b93-istiod-ca-cert\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:21:05.327367 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:05.327295    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f24gh"
Nov 01 05:21:05.327544 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:05.327297    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f24gh" event={"ID":"7758bc0a-0b7f-45d6-b688-53936075c3c7","Type":"ContainerDied","Data":"a644972937021369bebef3028d6ee8eb52d17090397ba953106d7ac4a7afe3b6"}
Nov 01 05:21:05.327544 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:05.327444    5689 scope.go:122] "RemoveContainer" containerID="05ba11e9854e456be39666e580103631670cdec4b09a850055b05a8a5f21d0ee"
Nov 01 05:21:05.328799 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:05.328700    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-2jwpv"
Nov 01 05:21:05.328799 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:05.328726    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-2jwpv" event={"ID":"5944a5e4-357d-4c82-930f-17b1f022f376","Type":"ContainerDied","Data":"08776ae7307041f825b1e091e6fd4d16ae8460387f597505a8cf4b7f1417c710"}
Nov 01 05:21:05.330319 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:05.330287    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jxzw" event={"ID":"54667545-c5d3-4893-9ade-e16c82cb5b93","Type":"ContainerDied","Data":"4c608f66ee39e3256288a4bbeae1817343a1ee77419961706169a68bcf15ece2"}
Nov 01 05:21:05.330468 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:05.330292    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jxzw"
Nov 01 05:21:05.331890 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:05.331869    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5t46"
Nov 01 05:21:05.331890 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:05.331876    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5t46" event={"ID":"231ce25c-5629-4780-9607-e5cd270afba5","Type":"ContainerDied","Data":"85671cd005352786bd17e184fbc1b6d314cb4b7233ddbc5acee2b099eba069ab"}
Nov 01 05:21:05.337885 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:05.337845    5689 scope.go:122] "RemoveContainer" containerID="472eba9828ecd5b8ab3b41248e35904b9176350ceab42b5d8ccc6207f5f2c50c"
Nov 01 05:21:05.348108 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:05.348082    5689 scope.go:122] "RemoveContainer" containerID="ff5e79c1f7e800789000628267819a6eceeee654a883f86558a39889f235a2e1"
Nov 01 05:21:05.355932 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:05.355917    5689 scope.go:122] "RemoveContainer" containerID="96c9d077aea787fd04dfd73dbba04c445ea921469b1f822a5db75386642ee5a0"
Nov 01 05:21:05.491859 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:05.491831    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5t46"]
Nov 01 05:21:05.541345 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:05.541315    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-h5t46"]
Nov 01 05:21:05.606839 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:05.606751    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jxzw"]
Nov 01 05:21:05.613947 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:05.613919    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4jxzw"]
Nov 01 05:21:05.718769 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:05.718742    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f24gh"]
Nov 01 05:21:05.723282 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:05.723258    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-f24gh"]
Nov 01 05:21:05.832716 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:05.832687    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-2jwpv"]
Nov 01 05:21:05.836342 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:05.836317    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-2jwpv"]
Nov 01 05:21:06.067203 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:06.067173    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="231ce25c-5629-4780-9607-e5cd270afba5" path="/var/lib/kubelet/pods/231ce25c-5629-4780-9607-e5cd270afba5/volumes"
Nov 01 05:21:06.067685 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:06.067671    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54667545-c5d3-4893-9ade-e16c82cb5b93" path="/var/lib/kubelet/pods/54667545-c5d3-4893-9ade-e16c82cb5b93/volumes"
Nov 01 05:21:06.068128 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:06.068111    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5944a5e4-357d-4c82-930f-17b1f022f376" path="/var/lib/kubelet/pods/5944a5e4-357d-4c82-930f-17b1f022f376/volumes"
Nov 01 05:21:06.068878 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:06.068863    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7758bc0a-0b7f-45d6-b688-53936075c3c7" path="/var/lib/kubelet/pods/7758bc0a-0b7f-45d6-b688-53936075c3c7/volumes"
Nov 01 05:21:08.408446 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:08.408416    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:21:09.884474 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:09.884429    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr"]
Nov 01 05:21:09.884918 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:09.884904    5689 cpu_manager.go:414] "containerMap: removing container" podUID="54667545-c5d3-4893-9ade-e16c82cb5b93" containerName="istio-proxy"
Nov 01 05:21:09.884972 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:09.884919    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54667545-c5d3-4893-9ade-e16c82cb5b93" containerName="istio-proxy"
Nov 01 05:21:09.884972 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:09.884934    5689 cpu_manager.go:414] "containerMap: removing container" podUID="7758bc0a-0b7f-45d6-b688-53936075c3c7" containerName="istio-proxy"
Nov 01 05:21:09.884972 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:09.884940    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7758bc0a-0b7f-45d6-b688-53936075c3c7" containerName="istio-proxy"
Nov 01 05:21:09.884972 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:09.884949    5689 cpu_manager.go:414] "containerMap: removing container" podUID="231ce25c-5629-4780-9607-e5cd270afba5" containerName="istio-proxy"
Nov 01 05:21:09.884972 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:09.884953    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="231ce25c-5629-4780-9607-e5cd270afba5" containerName="istio-proxy"
Nov 01 05:21:09.885149 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:09.884982    5689 cpu_manager.go:414] "containerMap: removing container" podUID="5944a5e4-357d-4c82-930f-17b1f022f376" containerName="istio-proxy"
Nov 01 05:21:09.885149 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:09.884988    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5944a5e4-357d-4c82-930f-17b1f022f376" containerName="istio-proxy"
Nov 01 05:21:09.885149 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:09.885053    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="5944a5e4-357d-4c82-930f-17b1f022f376" containerName="istio-proxy"
Nov 01 05:21:09.885149 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:09.885062    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="54667545-c5d3-4893-9ade-e16c82cb5b93" containerName="istio-proxy"
Nov 01 05:21:09.885149 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:09.885068    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="7758bc0a-0b7f-45d6-b688-53936075c3c7" containerName="istio-proxy"
Nov 01 05:21:09.885149 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:09.885074    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="231ce25c-5629-4780-9607-e5cd270afba5" containerName="istio-proxy"
Nov 01 05:21:09.890028 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:09.890011    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr"
Nov 01 05:21:09.996261 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:09.996233    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr"]
Nov 01 05:21:10.007112 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:10.007085    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91018a9a-6974-4492-8209-28c8c76ec6f7-istio-token\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr\" (UID: \"91018a9a-6974-4492-8209-28c8c76ec6f7\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr"
Nov 01 05:21:10.007316 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:10.007131    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91018a9a-6974-4492-8209-28c8c76ec6f7-istiod-ca-cert\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr\" (UID: \"91018a9a-6974-4492-8209-28c8c76ec6f7\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr"
Nov 01 05:21:10.007477 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:10.007385    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91018a9a-6974-4492-8209-28c8c76ec6f7-credential-socket\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr\" (UID: \"91018a9a-6974-4492-8209-28c8c76ec6f7\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr"
Nov 01 05:21:10.007477 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:10.007408    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91018a9a-6974-4492-8209-28c8c76ec6f7-istio-podinfo\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr\" (UID: \"91018a9a-6974-4492-8209-28c8c76ec6f7\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr"
Nov 01 05:21:10.007477 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:10.007447    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91018a9a-6974-4492-8209-28c8c76ec6f7-workload-socket\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr\" (UID: \"91018a9a-6974-4492-8209-28c8c76ec6f7\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr"
Nov 01 05:21:10.007683 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:10.007490    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91018a9a-6974-4492-8209-28c8c76ec6f7-workload-certs\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr\" (UID: \"91018a9a-6974-4492-8209-28c8c76ec6f7\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr"
Nov 01 05:21:10.007683 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:10.007521    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91018a9a-6974-4492-8209-28c8c76ec6f7-istio-data\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr\" (UID: \"91018a9a-6974-4492-8209-28c8c76ec6f7\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr"
Nov 01 05:21:10.007683 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:10.007567    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r85zj\" (UniqueName: \"kubernetes.io/projected/91018a9a-6974-4492-8209-28c8c76ec6f7-kube-api-access-r85zj\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr\" (UID: \"91018a9a-6974-4492-8209-28c8c76ec6f7\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr"
Nov 01 05:21:10.007683 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:10.007642    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91018a9a-6974-4492-8209-28c8c76ec6f7-istio-envoy\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr\" (UID: \"91018a9a-6974-4492-8209-28c8c76ec6f7\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr"
Nov 01 05:21:10.108688 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:10.108625    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91018a9a-6974-4492-8209-28c8c76ec6f7-istio-envoy\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr\" (UID: \"91018a9a-6974-4492-8209-28c8c76ec6f7\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr"
Nov 01 05:21:10.108882 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:10.108717    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91018a9a-6974-4492-8209-28c8c76ec6f7-istio-token\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr\" (UID: \"91018a9a-6974-4492-8209-28c8c76ec6f7\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr"
Nov 01 05:21:10.108882 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:10.108748    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91018a9a-6974-4492-8209-28c8c76ec6f7-istiod-ca-cert\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr\" (UID: \"91018a9a-6974-4492-8209-28c8c76ec6f7\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr"
Nov 01 05:21:10.108882 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:10.108788    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91018a9a-6974-4492-8209-28c8c76ec6f7-credential-socket\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr\" (UID: \"91018a9a-6974-4492-8209-28c8c76ec6f7\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr"
Nov 01 05:21:10.109047 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:10.108895    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91018a9a-6974-4492-8209-28c8c76ec6f7-istio-podinfo\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr\" (UID: \"91018a9a-6974-4492-8209-28c8c76ec6f7\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr"
Nov 01 05:21:10.109047 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:10.108936    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91018a9a-6974-4492-8209-28c8c76ec6f7-workload-socket\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr\" (UID: \"91018a9a-6974-4492-8209-28c8c76ec6f7\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr"
Nov 01 05:21:10.109047 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:10.108955    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91018a9a-6974-4492-8209-28c8c76ec6f7-workload-certs\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr\" (UID: \"91018a9a-6974-4492-8209-28c8c76ec6f7\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr"
Nov 01 05:21:10.109047 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:10.108988    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91018a9a-6974-4492-8209-28c8c76ec6f7-istio-data\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr\" (UID: \"91018a9a-6974-4492-8209-28c8c76ec6f7\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr"
Nov 01 05:21:10.109047 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:10.109038    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r85zj\" (UniqueName: \"kubernetes.io/projected/91018a9a-6974-4492-8209-28c8c76ec6f7-kube-api-access-r85zj\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr\" (UID: \"91018a9a-6974-4492-8209-28c8c76ec6f7\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr"
Nov 01 05:21:10.109737 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:10.109704    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91018a9a-6974-4492-8209-28c8c76ec6f7-credential-socket\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr\" (UID: \"91018a9a-6974-4492-8209-28c8c76ec6f7\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr"
Nov 01 05:21:10.109841 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:10.109784    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91018a9a-6974-4492-8209-28c8c76ec6f7-workload-certs\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr\" (UID: \"91018a9a-6974-4492-8209-28c8c76ec6f7\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr"
Nov 01 05:21:10.110000 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:10.109944    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91018a9a-6974-4492-8209-28c8c76ec6f7-istio-data\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr\" (UID: \"91018a9a-6974-4492-8209-28c8c76ec6f7\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr"
Nov 01 05:21:10.110000 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:10.109987    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91018a9a-6974-4492-8209-28c8c76ec6f7-istiod-ca-cert\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr\" (UID: \"91018a9a-6974-4492-8209-28c8c76ec6f7\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr"
Nov 01 05:21:10.110243 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:10.110079    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91018a9a-6974-4492-8209-28c8c76ec6f7-workload-socket\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr\" (UID: \"91018a9a-6974-4492-8209-28c8c76ec6f7\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr"
Nov 01 05:21:10.111823 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:10.111803    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91018a9a-6974-4492-8209-28c8c76ec6f7-istio-podinfo\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr\" (UID: \"91018a9a-6974-4492-8209-28c8c76ec6f7\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr"
Nov 01 05:21:10.112525 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:10.112507    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91018a9a-6974-4492-8209-28c8c76ec6f7-istio-envoy\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr\" (UID: \"91018a9a-6974-4492-8209-28c8c76ec6f7\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr"
Nov 01 05:21:10.169543 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:10.169505    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91018a9a-6974-4492-8209-28c8c76ec6f7-istio-token\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr\" (UID: \"91018a9a-6974-4492-8209-28c8c76ec6f7\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr"
Nov 01 05:21:10.169702 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:10.169648    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r85zj\" (UniqueName: \"kubernetes.io/projected/91018a9a-6974-4492-8209-28c8c76ec6f7-kube-api-access-r85zj\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr\" (UID: \"91018a9a-6974-4492-8209-28c8c76ec6f7\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr"
Nov 01 05:21:10.204641 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:10.204618    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr"
Nov 01 05:21:10.275395 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:10.274608    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:21:10.644975 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:10.644904    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr"]
Nov 01 05:21:10.648379 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:21:10.648356    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91018a9a_6974_4492_8209_28c8c76ec6f7.slice/crio-a88c78b8b3d0254f38ae4989112c3d5d09861ea3b688611d0d09ce5ff1b70610 WatchSource:0}: Error finding container a88c78b8b3d0254f38ae4989112c3d5d09861ea3b688611d0d09ce5ff1b70610: Status 404 returned error can't find the container with id a88c78b8b3d0254f38ae4989112c3d5d09861ea3b688611d0d09ce5ff1b70610
Nov 01 05:21:10.650866 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:10.650835    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:21:10.650981 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:10.650900    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:21:10.650981 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:10.650931    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:21:11.369988 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:11.369953    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr" event={"ID":"91018a9a-6974-4492-8209-28c8c76ec6f7","Type":"ContainerStarted","Data":"3fda437e1eedd5bee56af00e45a214d3e2c3a53a6c25b8d3618baf2126003694"}
Nov 01 05:21:11.370387 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:11.370003    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr" event={"ID":"91018a9a-6974-4492-8209-28c8c76ec6f7","Type":"ContainerStarted","Data":"a88c78b8b3d0254f38ae4989112c3d5d09861ea3b688611d0d09ce5ff1b70610"}
Nov 01 05:21:11.501480 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:11.500763    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr" podStartSLOduration=2.500744118 podStartE2EDuration="2.500744118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:21:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:21:11.495228235 +0000 UTC m=+1049.566024955" watchObservedRunningTime="2026-11-01 05:21:11.500744118 +0000 UTC m=+1049.571540840"
Nov 01 05:21:12.205267 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:12.205231    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr"
Nov 01 05:21:12.206722 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:12.206689    5689 patch_prober.go:29] interesting pod/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Nov 01 05:21:12.206869 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:12.206745    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr" podUID="91018a9a-6974-4492-8209-28c8c76ec6f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Nov 01 05:21:12.924523 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:12.924487    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4dbthwg"]
Nov 01 05:21:13.206062 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:13.205979    5689 patch_prober.go:29] interesting pod/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Nov 01 05:21:13.206217 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:13.206058    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr" podUID="91018a9a-6974-4492-8209-28c8c76ec6f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Nov 01 05:21:13.271186 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:13.271156    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:21:14.206032 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:14.205994    5689 patch_prober.go:29] interesting pod/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Nov 01 05:21:14.206550 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:14.206059    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr" podUID="91018a9a-6974-4492-8209-28c8c76ec6f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Nov 01 05:21:15.205231 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:15.205192    5689 patch_prober.go:29] interesting pod/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Nov 01 05:21:15.205407 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:15.205272    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr" podUID="91018a9a-6974-4492-8209-28c8c76ec6f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Nov 01 05:21:15.947351 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:15.947319    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:21:16.135755 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:16.135722    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:21:16.206030 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:16.205490    5689 patch_prober.go:29] interesting pod/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Nov 01 05:21:16.206030 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:16.205574    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr" podUID="91018a9a-6974-4492-8209-28c8c76ec6f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Nov 01 05:21:17.205536 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:17.205496    5689 patch_prober.go:29] interesting pod/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Nov 01 05:21:17.205921 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:17.205561    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr" podUID="91018a9a-6974-4492-8209-28c8c76ec6f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Nov 01 05:21:17.925703 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:17.925650    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4dbthwg" podUID="6d149758-c12f-4374-80ac-9da5bbea5071" containerName="payload-processing" containerID="cri-o://9e137522254d8a0c07d63dbd78dba610a35ccc63212e68520806625250572e7f" gracePeriod=25
Nov 01 05:21:18.182618 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:18.182549    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4dbthwg"
Nov 01 05:21:18.210239 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:18.210213    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr"
Nov 01 05:21:18.210614 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:18.210482    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr"
Nov 01 05:21:18.211199 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:18.211180    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr"
Nov 01 05:21:18.221957 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:18.221933    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:21:18.233052 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:18.233025    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d149758-c12f-4374-80ac-9da5bbea5071-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6d149758-c12f-4374-80ac-9da5bbea5071-routing-overlay-volume\") pod \"6d149758-c12f-4374-80ac-9da5bbea5071\" (UID: \"6d149758-c12f-4374-80ac-9da5bbea5071\") "
Nov 01 05:21:18.233155 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:18.233073    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d149758-c12f-4374-80ac-9da5bbea5071-kube-api-access-vgsg6\" (UniqueName: \"kubernetes.io/projected/6d149758-c12f-4374-80ac-9da5bbea5071-kube-api-access-vgsg6\") pod \"6d149758-c12f-4374-80ac-9da5bbea5071\" (UID: \"6d149758-c12f-4374-80ac-9da5bbea5071\") "
Nov 01 05:21:18.233209 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:18.233174    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d149758-c12f-4374-80ac-9da5bbea5071-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6d149758-c12f-4374-80ac-9da5bbea5071-provider-credentials\") pod \"6d149758-c12f-4374-80ac-9da5bbea5071\" (UID: \"6d149758-c12f-4374-80ac-9da5bbea5071\") "
Nov 01 05:21:18.233291 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:18.233275    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d149758-c12f-4374-80ac-9da5bbea5071-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d149758-c12f-4374-80ac-9da5bbea5071-plugins-config-volume\") pod \"6d149758-c12f-4374-80ac-9da5bbea5071\" (UID: \"6d149758-c12f-4374-80ac-9da5bbea5071\") "
Nov 01 05:21:18.233441 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:18.233419    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d149758-c12f-4374-80ac-9da5bbea5071-routing-overlay-volume" pod "6d149758-c12f-4374-80ac-9da5bbea5071" (UID: "6d149758-c12f-4374-80ac-9da5bbea5071"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:21:18.233790 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:18.233694    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d149758-c12f-4374-80ac-9da5bbea5071-plugins-config-volume" pod "6d149758-c12f-4374-80ac-9da5bbea5071" (UID: "6d149758-c12f-4374-80ac-9da5bbea5071"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:21:18.233790 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:18.233721    5689 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6d149758-c12f-4374-80ac-9da5bbea5071-routing-overlay-volume\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:21:18.235641 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:18.235613    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d149758-c12f-4374-80ac-9da5bbea5071-kube-api-access-vgsg6" pod "6d149758-c12f-4374-80ac-9da5bbea5071" (UID: "6d149758-c12f-4374-80ac-9da5bbea5071"). InnerVolumeSpecName "kube-api-access-vgsg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:21:18.235850 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:18.235827    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d149758-c12f-4374-80ac-9da5bbea5071-provider-credentials" pod "6d149758-c12f-4374-80ac-9da5bbea5071" (UID: "6d149758-c12f-4374-80ac-9da5bbea5071"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:21:18.334544 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:18.334512    5689 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6d149758-c12f-4374-80ac-9da5bbea5071-provider-credentials\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:21:18.334756 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:18.334554    5689 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d149758-c12f-4374-80ac-9da5bbea5071-plugins-config-volume\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:21:18.334756 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:18.334571    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgsg6\" (UniqueName: \"kubernetes.io/projected/6d149758-c12f-4374-80ac-9da5bbea5071-kube-api-access-vgsg6\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:21:18.440910 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:18.440630    5689 generic.go:356] "Generic (PLEG): container finished" podID="6d149758-c12f-4374-80ac-9da5bbea5071" containerID="9e137522254d8a0c07d63dbd78dba610a35ccc63212e68520806625250572e7f" exitCode=0
Nov 01 05:21:18.441880 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:18.441840    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4dbthwg" event={"ID":"6d149758-c12f-4374-80ac-9da5bbea5071","Type":"ContainerDied","Data":"9e137522254d8a0c07d63dbd78dba610a35ccc63212e68520806625250572e7f"}
Nov 01 05:21:18.442006 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:18.441944    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4dbthwg" event={"ID":"6d149758-c12f-4374-80ac-9da5bbea5071","Type":"ContainerDied","Data":"af7fc51da28853459bb5ca5664841d64959a3d8251a0163716fcc0596edc7174"}
Nov 01 05:21:18.442006 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:18.441975    5689 scope.go:122] "RemoveContainer" containerID="9e137522254d8a0c07d63dbd78dba610a35ccc63212e68520806625250572e7f"
Nov 01 05:21:18.442770 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:18.442744    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4dbthwg"
Nov 01 05:21:18.453717 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:18.453698    5689 scope.go:122] "RemoveContainer" containerID="9e137522254d8a0c07d63dbd78dba610a35ccc63212e68520806625250572e7f"
Nov 01 05:21:18.454180 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:21:18.454159    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e137522254d8a0c07d63dbd78dba610a35ccc63212e68520806625250572e7f\": container with ID starting with 9e137522254d8a0c07d63dbd78dba610a35ccc63212e68520806625250572e7f not found: ID does not exist" containerID="9e137522254d8a0c07d63dbd78dba610a35ccc63212e68520806625250572e7f"
Nov 01 05:21:18.454267 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:18.454192    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e137522254d8a0c07d63dbd78dba610a35ccc63212e68520806625250572e7f"} err="failed to get container status \"9e137522254d8a0c07d63dbd78dba610a35ccc63212e68520806625250572e7f\": rpc error: code = NotFound desc = could not find container \"9e137522254d8a0c07d63dbd78dba610a35ccc63212e68520806625250572e7f\": container with ID starting with 9e137522254d8a0c07d63dbd78dba610a35ccc63212e68520806625250572e7f not found: ID does not exist"
Nov 01 05:21:18.607914 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:18.607883    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4dbthwg"]
Nov 01 05:21:18.614973 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:18.614949    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-9c4dbthwg"]
Nov 01 05:21:19.940382 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:19.940344    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6nz2mn"]
Nov 01 05:21:19.941027 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:19.941006    5689 cpu_manager.go:414] "containerMap: removing container" podUID="6d149758-c12f-4374-80ac-9da5bbea5071" containerName="payload-processing"
Nov 01 05:21:19.941119 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:19.941030    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d149758-c12f-4374-80ac-9da5bbea5071" containerName="payload-processing"
Nov 01 05:21:19.941233 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:19.941218    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d149758-c12f-4374-80ac-9da5bbea5071" containerName="payload-processing"
Nov 01 05:21:19.945607 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:19.945584    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6nz2mn"
Nov 01 05:21:19.998079 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:19.998052    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2fd4c9\"/\"openshift-service-ca.crt\""
Nov 01 05:21:19.998079 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:19.998060    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2fd4c9\"/\"routing-overlay\""
Nov 01 05:21:20.006348 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:20.006325    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2fd4c9\"/\"agc-cred-e326123de79d\""
Nov 01 05:21:20.006508 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:20.006365    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2fd4c9\"/\"agc-cred-f8c698a71031\""
Nov 01 05:21:20.047131 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:20.047106    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2fd4c9\"/\"payload-processing-external-model-plugins-e2e-extmod-b-2fd4c9\""
Nov 01 05:21:20.049004 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:20.048985    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6nz2mn"]
Nov 01 05:21:20.054031 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:20.054009    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqbpw\" (UniqueName: \"kubernetes.io/projected/10e665b2-7fff-4260-9626-f7a834606eef-kube-api-access-nqbpw\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6nz2mn\" (UID: \"10e665b2-7fff-4260-9626-f7a834606eef\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6nz2mn"
Nov 01 05:21:20.054124 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:20.054048    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10e665b2-7fff-4260-9626-f7a834606eef-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6nz2mn\" (UID: \"10e665b2-7fff-4260-9626-f7a834606eef\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6nz2mn"
Nov 01 05:21:20.054124 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:20.054078    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/10e665b2-7fff-4260-9626-f7a834606eef-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6nz2mn\" (UID: \"10e665b2-7fff-4260-9626-f7a834606eef\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6nz2mn"
Nov 01 05:21:20.054208 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:20.054166    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/10e665b2-7fff-4260-9626-f7a834606eef-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6nz2mn\" (UID: \"10e665b2-7fff-4260-9626-f7a834606eef\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6nz2mn"
Nov 01 05:21:20.067992 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:20.067959    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d149758-c12f-4374-80ac-9da5bbea5071" path="/var/lib/kubelet/pods/6d149758-c12f-4374-80ac-9da5bbea5071/volumes"
Nov 01 05:21:20.098604 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:20.098582    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2fd4c9\"/\"kube-root-ca.crt\""
Nov 01 05:21:20.098690 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:20.098644    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2fd4c9\"/\"agc-cred-17306a2b8c16\""
Nov 01 05:21:20.098736 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:20.098723    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2fd4c9\"/\"payload-processing-external-model-e2e--928c4e32-dockercfg-r25vt\""
Nov 01 05:21:20.155075 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:20.155051    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqbpw\" (UniqueName: \"kubernetes.io/projected/10e665b2-7fff-4260-9626-f7a834606eef-kube-api-access-nqbpw\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6nz2mn\" (UID: \"10e665b2-7fff-4260-9626-f7a834606eef\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6nz2mn"
Nov 01 05:21:20.155170 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:20.155081    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10e665b2-7fff-4260-9626-f7a834606eef-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6nz2mn\" (UID: \"10e665b2-7fff-4260-9626-f7a834606eef\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6nz2mn"
Nov 01 05:21:20.155170 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:20.155100    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/10e665b2-7fff-4260-9626-f7a834606eef-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6nz2mn\" (UID: \"10e665b2-7fff-4260-9626-f7a834606eef\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6nz2mn"
Nov 01 05:21:20.155170 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:20.155126    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/10e665b2-7fff-4260-9626-f7a834606eef-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6nz2mn\" (UID: \"10e665b2-7fff-4260-9626-f7a834606eef\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6nz2mn"
Nov 01 05:21:20.155885 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:20.155864    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/10e665b2-7fff-4260-9626-f7a834606eef-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6nz2mn\" (UID: \"10e665b2-7fff-4260-9626-f7a834606eef\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6nz2mn"
Nov 01 05:21:20.155977 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:20.155961    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10e665b2-7fff-4260-9626-f7a834606eef-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6nz2mn\" (UID: \"10e665b2-7fff-4260-9626-f7a834606eef\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6nz2mn"
Nov 01 05:21:20.157811 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:20.157791    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/10e665b2-7fff-4260-9626-f7a834606eef-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6nz2mn\" (UID: \"10e665b2-7fff-4260-9626-f7a834606eef\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6nz2mn"
Nov 01 05:21:20.216063 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:20.216001    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqbpw\" (UniqueName: \"kubernetes.io/projected/10e665b2-7fff-4260-9626-f7a834606eef-kube-api-access-nqbpw\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6nz2mn\" (UID: \"10e665b2-7fff-4260-9626-f7a834606eef\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6nz2mn"
Nov 01 05:21:20.258460 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:20.258435    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6nz2mn"
Nov 01 05:21:20.637733 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:20.637708    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6nz2mn"]
Nov 01 05:21:20.639975 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:21:20.639943    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10e665b2_7fff_4260_9626_f7a834606eef.slice/crio-92d95367f45cdbe50f01d67c7462666082e9a64c1495626fe02f7f4cf21539fb WatchSource:0}: Error finding container 92d95367f45cdbe50f01d67c7462666082e9a64c1495626fe02f7f4cf21539fb: Status 404 returned error can't find the container with id 92d95367f45cdbe50f01d67c7462666082e9a64c1495626fe02f7f4cf21539fb
Nov 01 05:21:21.461103 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:21.461068    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6nz2mn" event={"ID":"10e665b2-7fff-4260-9626-f7a834606eef","Type":"ContainerStarted","Data":"862a0eedae263cf4efe90f4044a05fb34f90a2833e98d5d83dc7d8422b6331fc"}
Nov 01 05:21:21.461563 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:21.461119    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6nz2mn" event={"ID":"10e665b2-7fff-4260-9626-f7a834606eef","Type":"ContainerStarted","Data":"92d95367f45cdbe50f01d67c7462666082e9a64c1495626fe02f7f4cf21539fb"}
Nov 01 05:21:21.461563 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:21.461199    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6nz2mn"
Nov 01 05:21:21.578073 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:21.578025    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6nz2mn" podStartSLOduration=2.578012291 podStartE2EDuration="2.578012291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:21:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:21:21.577080252 +0000 UTC m=+1059.647876972" watchObservedRunningTime="2026-11-01 05:21:21.578012291 +0000 UTC m=+1059.648809010"
Nov 01 05:21:23.058623 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:23.058586    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:21:23.923260 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:23.923229    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-sgdbb"]
Nov 01 05:21:23.928135 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:23.928117    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-sgdbb"
Nov 01 05:21:23.998694 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:23.998666    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3654852a-2923-46d4-a244-8a907545ed9c-credential-socket\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-sgdbb\" (UID: \"3654852a-2923-46d4-a244-8a907545ed9c\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-sgdbb"
Nov 01 05:21:23.998831 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:23.998723    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3654852a-2923-46d4-a244-8a907545ed9c-istio-token\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-sgdbb\" (UID: \"3654852a-2923-46d4-a244-8a907545ed9c\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-sgdbb"
Nov 01 05:21:23.998831 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:23.998816    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3654852a-2923-46d4-a244-8a907545ed9c-istiod-ca-cert\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-sgdbb\" (UID: \"3654852a-2923-46d4-a244-8a907545ed9c\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-sgdbb"
Nov 01 05:21:23.998914 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:23.998838    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3654852a-2923-46d4-a244-8a907545ed9c-workload-socket\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-sgdbb\" (UID: \"3654852a-2923-46d4-a244-8a907545ed9c\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-sgdbb"
Nov 01 05:21:23.998914 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:23.998870    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3654852a-2923-46d4-a244-8a907545ed9c-workload-certs\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-sgdbb\" (UID: \"3654852a-2923-46d4-a244-8a907545ed9c\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-sgdbb"
Nov 01 05:21:23.999017 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:23.998990    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3654852a-2923-46d4-a244-8a907545ed9c-istio-podinfo\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-sgdbb\" (UID: \"3654852a-2923-46d4-a244-8a907545ed9c\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-sgdbb"
Nov 01 05:21:23.999071 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:23.999052    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb94c\" (UniqueName: \"kubernetes.io/projected/3654852a-2923-46d4-a244-8a907545ed9c-kube-api-access-fb94c\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-sgdbb\" (UID: \"3654852a-2923-46d4-a244-8a907545ed9c\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-sgdbb"
Nov 01 05:21:23.999136 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:23.999121    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3654852a-2923-46d4-a244-8a907545ed9c-istio-envoy\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-sgdbb\" (UID: \"3654852a-2923-46d4-a244-8a907545ed9c\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-sgdbb"
Nov 01 05:21:23.999183 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:23.999162    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3654852a-2923-46d4-a244-8a907545ed9c-istio-data\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-sgdbb\" (UID: \"3654852a-2923-46d4-a244-8a907545ed9c\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-sgdbb"
Nov 01 05:21:24.031775 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:24.031748    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-sgdbb"]
Nov 01 05:21:24.100215 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:24.100186    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3654852a-2923-46d4-a244-8a907545ed9c-credential-socket\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-sgdbb\" (UID: \"3654852a-2923-46d4-a244-8a907545ed9c\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-sgdbb"
Nov 01 05:21:24.100693 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:24.100236    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3654852a-2923-46d4-a244-8a907545ed9c-istio-token\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-sgdbb\" (UID: \"3654852a-2923-46d4-a244-8a907545ed9c\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-sgdbb"
Nov 01 05:21:24.100693 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:24.100273    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3654852a-2923-46d4-a244-8a907545ed9c-istiod-ca-cert\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-sgdbb\" (UID: \"3654852a-2923-46d4-a244-8a907545ed9c\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-sgdbb"
Nov 01 05:21:24.100693 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:24.100304    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3654852a-2923-46d4-a244-8a907545ed9c-workload-socket\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-sgdbb\" (UID: \"3654852a-2923-46d4-a244-8a907545ed9c\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-sgdbb"
Nov 01 05:21:24.100693 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:24.100482    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3654852a-2923-46d4-a244-8a907545ed9c-workload-certs\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-sgdbb\" (UID: \"3654852a-2923-46d4-a244-8a907545ed9c\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-sgdbb"
Nov 01 05:21:24.100693 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:24.100557    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3654852a-2923-46d4-a244-8a907545ed9c-istio-podinfo\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-sgdbb\" (UID: \"3654852a-2923-46d4-a244-8a907545ed9c\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-sgdbb"
Nov 01 05:21:24.100693 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:24.100599    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb94c\" (UniqueName: \"kubernetes.io/projected/3654852a-2923-46d4-a244-8a907545ed9c-kube-api-access-fb94c\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-sgdbb\" (UID: \"3654852a-2923-46d4-a244-8a907545ed9c\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-sgdbb"
Nov 01 05:21:24.100693 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:24.100660    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3654852a-2923-46d4-a244-8a907545ed9c-credential-socket\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-sgdbb\" (UID: \"3654852a-2923-46d4-a244-8a907545ed9c\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-sgdbb"
Nov 01 05:21:24.100693 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:24.100687    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3654852a-2923-46d4-a244-8a907545ed9c-istio-envoy\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-sgdbb\" (UID: \"3654852a-2923-46d4-a244-8a907545ed9c\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-sgdbb"
Nov 01 05:21:24.101180 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:24.100713    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3654852a-2923-46d4-a244-8a907545ed9c-istio-data\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-sgdbb\" (UID: \"3654852a-2923-46d4-a244-8a907545ed9c\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-sgdbb"
Nov 01 05:21:24.101180 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:24.100731    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3654852a-2923-46d4-a244-8a907545ed9c-workload-socket\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-sgdbb\" (UID: \"3654852a-2923-46d4-a244-8a907545ed9c\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-sgdbb"
Nov 01 05:21:24.101180 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:24.100810    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3654852a-2923-46d4-a244-8a907545ed9c-workload-certs\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-sgdbb\" (UID: \"3654852a-2923-46d4-a244-8a907545ed9c\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-sgdbb"
Nov 01 05:21:24.101180 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:24.101064    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3654852a-2923-46d4-a244-8a907545ed9c-istio-data\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-sgdbb\" (UID: \"3654852a-2923-46d4-a244-8a907545ed9c\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-sgdbb"
Nov 01 05:21:24.101180 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:24.101119    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3654852a-2923-46d4-a244-8a907545ed9c-istiod-ca-cert\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-sgdbb\" (UID: \"3654852a-2923-46d4-a244-8a907545ed9c\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-sgdbb"
Nov 01 05:21:24.102949 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:24.102921    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3654852a-2923-46d4-a244-8a907545ed9c-istio-envoy\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-sgdbb\" (UID: \"3654852a-2923-46d4-a244-8a907545ed9c\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-sgdbb"
Nov 01 05:21:24.103320 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:24.103297    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3654852a-2923-46d4-a244-8a907545ed9c-istio-podinfo\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-sgdbb\" (UID: \"3654852a-2923-46d4-a244-8a907545ed9c\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-sgdbb"
Nov 01 05:21:24.160767 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:24.160741    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3654852a-2923-46d4-a244-8a907545ed9c-istio-token\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-sgdbb\" (UID: \"3654852a-2923-46d4-a244-8a907545ed9c\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-sgdbb"
Nov 01 05:21:24.160911 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:24.160881    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb94c\" (UniqueName: \"kubernetes.io/projected/3654852a-2923-46d4-a244-8a907545ed9c-kube-api-access-fb94c\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-sgdbb\" (UID: \"3654852a-2923-46d4-a244-8a907545ed9c\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-sgdbb"
Nov 01 05:21:24.241921 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:24.241844    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-sgdbb"
Nov 01 05:21:24.673388 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:24.673273    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-sgdbb"]
Nov 01 05:21:24.676140 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:21:24.676114    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3654852a_2923_46d4_a244_8a907545ed9c.slice/crio-326d0f60ab833bd9980e10c231569a040244f7a0671dab0615957bc3b75b0b93 WatchSource:0}: Error finding container 326d0f60ab833bd9980e10c231569a040244f7a0671dab0615957bc3b75b0b93: Status 404 returned error can't find the container with id 326d0f60ab833bd9980e10c231569a040244f7a0671dab0615957bc3b75b0b93
Nov 01 05:21:24.678440 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:24.678412    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:21:24.678515 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:24.678493    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:21:24.678556 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:24.678521    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:21:25.164623 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:25.164591    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:21:25.503161 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:25.503075    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-sgdbb" event={"ID":"3654852a-2923-46d4-a244-8a907545ed9c","Type":"ContainerStarted","Data":"583288b55e7c8e16beb20b204914acb34e85591bd8e203af8856448a4f40d826"}
Nov 01 05:21:25.503366 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:25.503344    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-sgdbb" event={"ID":"3654852a-2923-46d4-a244-8a907545ed9c","Type":"ContainerStarted","Data":"326d0f60ab833bd9980e10c231569a040244f7a0671dab0615957bc3b75b0b93"}
Nov 01 05:21:25.625235 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:25.625177    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-sgdbb" podStartSLOduration=2.625162225 podStartE2EDuration="2.625162225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:21:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:21:25.62454825 +0000 UTC m=+1063.695344970" watchObservedRunningTime="2026-11-01 05:21:25.625162225 +0000 UTC m=+1063.695958945"
Nov 01 05:21:26.242939 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:26.242898    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-sgdbb"
Nov 01 05:21:26.244257 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:26.244230    5689 patch_prober.go:29] interesting pod/e2e-worker-w4-a48099-openshift-default-577864759-sgdbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 01 05:21:26.244376 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:26.244293    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-sgdbb" podUID="3654852a-2923-46d4-a244-8a907545ed9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 01 05:21:26.269007 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:26.268977    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-a12c35/payload-processing-external-model-e2e-extmod-a-a12c35-664fd8jf5"]
Nov 01 05:21:26.272822 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:26.272804    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a12c35/payload-processing-external-model-e2e-extmod-a-a12c35-664fd8jf5"
Nov 01 05:21:26.325109 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:26.325068    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a12c35\"/\"agc-cred-6b501b136f59\""
Nov 01 05:21:26.325246 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:26.325128    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a12c35\"/\"routing-overlay\""
Nov 01 05:21:26.325246 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:26.325157    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a12c35\"/\"payload-processing-external-model-plugins-e2e-extmod-a-a12c35\""
Nov 01 05:21:26.325246 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:26.325182    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a12c35\"/\"openshift-service-ca.crt\""
Nov 01 05:21:26.325246 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:26.325195    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a12c35\"/\"payload-processing-external-model-e2e--57fe8cc0-dockercfg-jp2f8\""
Nov 01 05:21:26.325246 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:26.325187    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a12c35\"/\"kube-root-ca.crt\""
Nov 01 05:21:26.357982 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:26.357952    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a12c35/payload-processing-external-model-e2e-extmod-a-a12c35-664fd8jf5"]
Nov 01 05:21:26.445863 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:26.445831    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e5f6593b-d8a7-434e-bd2a-4555aac39916-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a12c35-664fd8jf5\" (UID: \"e5f6593b-d8a7-434e-bd2a-4555aac39916\") " pod="ai-tenant-e2e-extmod-a-a12c35/payload-processing-external-model-e2e-extmod-a-a12c35-664fd8jf5"
Nov 01 05:21:26.446059 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:26.445898    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5f6593b-d8a7-434e-bd2a-4555aac39916-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a12c35-664fd8jf5\" (UID: \"e5f6593b-d8a7-434e-bd2a-4555aac39916\") " pod="ai-tenant-e2e-extmod-a-a12c35/payload-processing-external-model-e2e-extmod-a-a12c35-664fd8jf5"
Nov 01 05:21:26.446059 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:26.446034    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrqrc\" (UniqueName: \"kubernetes.io/projected/e5f6593b-d8a7-434e-bd2a-4555aac39916-kube-api-access-vrqrc\") pod \"payload-processing-external-model-e2e-extmod-a-a12c35-664fd8jf5\" (UID: \"e5f6593b-d8a7-434e-bd2a-4555aac39916\") " pod="ai-tenant-e2e-extmod-a-a12c35/payload-processing-external-model-e2e-extmod-a-a12c35-664fd8jf5"
Nov 01 05:21:26.446166 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:26.446069    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e5f6593b-d8a7-434e-bd2a-4555aac39916-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a12c35-664fd8jf5\" (UID: \"e5f6593b-d8a7-434e-bd2a-4555aac39916\") " pod="ai-tenant-e2e-extmod-a-a12c35/payload-processing-external-model-e2e-extmod-a-a12c35-664fd8jf5"
Nov 01 05:21:26.547094 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:26.547010    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5f6593b-d8a7-434e-bd2a-4555aac39916-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a12c35-664fd8jf5\" (UID: \"e5f6593b-d8a7-434e-bd2a-4555aac39916\") " pod="ai-tenant-e2e-extmod-a-a12c35/payload-processing-external-model-e2e-extmod-a-a12c35-664fd8jf5"
Nov 01 05:21:26.547248 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:26.547109    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrqrc\" (UniqueName: \"kubernetes.io/projected/e5f6593b-d8a7-434e-bd2a-4555aac39916-kube-api-access-vrqrc\") pod \"payload-processing-external-model-e2e-extmod-a-a12c35-664fd8jf5\" (UID: \"e5f6593b-d8a7-434e-bd2a-4555aac39916\") " pod="ai-tenant-e2e-extmod-a-a12c35/payload-processing-external-model-e2e-extmod-a-a12c35-664fd8jf5"
Nov 01 05:21:26.547248 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:26.547147    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e5f6593b-d8a7-434e-bd2a-4555aac39916-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a12c35-664fd8jf5\" (UID: \"e5f6593b-d8a7-434e-bd2a-4555aac39916\") " pod="ai-tenant-e2e-extmod-a-a12c35/payload-processing-external-model-e2e-extmod-a-a12c35-664fd8jf5"
Nov 01 05:21:26.547248 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:26.547206    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e5f6593b-d8a7-434e-bd2a-4555aac39916-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a12c35-664fd8jf5\" (UID: \"e5f6593b-d8a7-434e-bd2a-4555aac39916\") " pod="ai-tenant-e2e-extmod-a-a12c35/payload-processing-external-model-e2e-extmod-a-a12c35-664fd8jf5"
Nov 01 05:21:26.547372 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:21:26.547290    5689 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-a12c35/agc-cred-6b501b136f59: secret "agc-cred-6b501b136f59" not found
Nov 01 05:21:26.547372 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:21:26.547309    5689 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-a12c35/payload-processing-external-model-e2e-extmod-a-a12c35-664fd8jf5: secret "agc-cred-6b501b136f59" not found
Nov 01 05:21:26.547448 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:21:26.547376    5689 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e5f6593b-d8a7-434e-bd2a-4555aac39916-provider-credentials podName:e5f6593b-d8a7-434e-bd2a-4555aac39916 nodeName:}" failed. No retries permitted until 2026-11-01 05:21:27.047361091 +0000 UTC m=+1065.118157807 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e5f6593b-d8a7-434e-bd2a-4555aac39916-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-a12c35-664fd8jf5" (UID: "e5f6593b-d8a7-434e-bd2a-4555aac39916") : secret "agc-cred-6b501b136f59" not found
Nov 01 05:21:26.547905 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:26.547886    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5f6593b-d8a7-434e-bd2a-4555aac39916-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a12c35-664fd8jf5\" (UID: \"e5f6593b-d8a7-434e-bd2a-4555aac39916\") " pod="ai-tenant-e2e-extmod-a-a12c35/payload-processing-external-model-e2e-extmod-a-a12c35-664fd8jf5"
Nov 01 05:21:26.547945 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:26.547886    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e5f6593b-d8a7-434e-bd2a-4555aac39916-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a12c35-664fd8jf5\" (UID: \"e5f6593b-d8a7-434e-bd2a-4555aac39916\") " pod="ai-tenant-e2e-extmod-a-a12c35/payload-processing-external-model-e2e-extmod-a-a12c35-664fd8jf5"
Nov 01 05:21:26.610331 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:26.610296    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrqrc\" (UniqueName: \"kubernetes.io/projected/e5f6593b-d8a7-434e-bd2a-4555aac39916-kube-api-access-vrqrc\") pod \"payload-processing-external-model-e2e-extmod-a-a12c35-664fd8jf5\" (UID: \"e5f6593b-d8a7-434e-bd2a-4555aac39916\") " pod="ai-tenant-e2e-extmod-a-a12c35/payload-processing-external-model-e2e-extmod-a-a12c35-664fd8jf5"
Nov 01 05:21:27.053293 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:27.053259    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e5f6593b-d8a7-434e-bd2a-4555aac39916-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a12c35-664fd8jf5\" (UID: \"e5f6593b-d8a7-434e-bd2a-4555aac39916\") " pod="ai-tenant-e2e-extmod-a-a12c35/payload-processing-external-model-e2e-extmod-a-a12c35-664fd8jf5"
Nov 01 05:21:27.053494 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:21:27.053403    5689 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-a12c35/agc-cred-6b501b136f59: secret "agc-cred-6b501b136f59" not found
Nov 01 05:21:27.053494 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:21:27.053421    5689 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-a12c35/payload-processing-external-model-e2e-extmod-a-a12c35-664fd8jf5: secret "agc-cred-6b501b136f59" not found
Nov 01 05:21:27.053599 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:21:27.053498    5689 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e5f6593b-d8a7-434e-bd2a-4555aac39916-provider-credentials podName:e5f6593b-d8a7-434e-bd2a-4555aac39916 nodeName:}" failed. No retries permitted until 2026-11-01 05:21:28.053484096 +0000 UTC m=+1066.124280797 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e5f6593b-d8a7-434e-bd2a-4555aac39916-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-a12c35-664fd8jf5" (UID: "e5f6593b-d8a7-434e-bd2a-4555aac39916") : secret "agc-cred-6b501b136f59" not found
Nov 01 05:21:27.242211 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:27.242180    5689 patch_prober.go:29] interesting pod/e2e-worker-w4-a48099-openshift-default-577864759-sgdbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 01 05:21:27.242377 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:27.242242    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-sgdbb" podUID="3654852a-2923-46d4-a244-8a907545ed9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 01 05:21:27.900085 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:27.899987    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:21:28.064620 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:28.064593    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e5f6593b-d8a7-434e-bd2a-4555aac39916-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a12c35-664fd8jf5\" (UID: \"e5f6593b-d8a7-434e-bd2a-4555aac39916\") " pod="ai-tenant-e2e-extmod-a-a12c35/payload-processing-external-model-e2e-extmod-a-a12c35-664fd8jf5"
Nov 01 05:21:28.064776 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:21:28.064723    5689 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-a12c35/agc-cred-6b501b136f59: secret "agc-cred-6b501b136f59" not found
Nov 01 05:21:28.064776 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:21:28.064736    5689 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-a12c35/payload-processing-external-model-e2e-extmod-a-a12c35-664fd8jf5: secret "agc-cred-6b501b136f59" not found
Nov 01 05:21:28.064851 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:21:28.064796    5689 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e5f6593b-d8a7-434e-bd2a-4555aac39916-provider-credentials podName:e5f6593b-d8a7-434e-bd2a-4555aac39916 nodeName:}" failed. No retries permitted until 2026-11-01 05:21:30.06478264 +0000 UTC m=+1068.135579338 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e5f6593b-d8a7-434e-bd2a-4555aac39916-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-a12c35-664fd8jf5" (UID: "e5f6593b-d8a7-434e-bd2a-4555aac39916") : secret "agc-cred-6b501b136f59" not found
Nov 01 05:21:28.243155 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:28.243076    5689 patch_prober.go:29] interesting pod/e2e-worker-w4-a48099-openshift-default-577864759-sgdbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 01 05:21:28.243155 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:28.243139    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-sgdbb" podUID="3654852a-2923-46d4-a244-8a907545ed9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 01 05:21:29.242369 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:29.242331    5689 patch_prober.go:29] interesting pod/e2e-worker-w4-a48099-openshift-default-577864759-sgdbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 01 05:21:29.242790 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:29.242397    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-sgdbb" podUID="3654852a-2923-46d4-a244-8a907545ed9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 01 05:21:30.070325 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:21:30.070292    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:21:30.111989 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:30.111956    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e5f6593b-d8a7-434e-bd2a-4555aac39916-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a12c35-664fd8jf5\" (UID: \"e5f6593b-d8a7-434e-bd2a-4555aac39916\") " pod="ai-tenant-e2e-extmod-a-a12c35/payload-processing-external-model-e2e-extmod-a-a12c35-664fd8jf5"
Nov 01 05:21:30.112186 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:21:30.112101    5689 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-a12c35/agc-cred-6b501b136f59: secret "agc-cred-6b501b136f59" not found
Nov 01 05:21:30.112186 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:21:30.112121    5689 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-a12c35/payload-processing-external-model-e2e-extmod-a-a12c35-664fd8jf5: secret "agc-cred-6b501b136f59" not found
Nov 01 05:21:30.112186 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:21:30.112173    5689 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e5f6593b-d8a7-434e-bd2a-4555aac39916-provider-credentials podName:e5f6593b-d8a7-434e-bd2a-4555aac39916 nodeName:}" failed. No retries permitted until 2026-11-01 05:21:34.112160593 +0000 UTC m=+1072.182957295 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e5f6593b-d8a7-434e-bd2a-4555aac39916-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-a12c35-664fd8jf5" (UID: "e5f6593b-d8a7-434e-bd2a-4555aac39916") : secret "agc-cred-6b501b136f59" not found
Nov 01 05:21:30.242572 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:30.242531    5689 patch_prober.go:29] interesting pod/e2e-worker-w4-a48099-openshift-default-577864759-sgdbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 01 05:21:30.243041 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:30.242593    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-sgdbb" podUID="3654852a-2923-46d4-a244-8a907545ed9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 01 05:21:31.242739 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:31.242707    5689 patch_prober.go:29] interesting pod/e2e-worker-w4-a48099-openshift-default-577864759-sgdbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 01 05:21:31.243127 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:31.242759    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-sgdbb" podUID="3654852a-2923-46d4-a244-8a907545ed9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 01 05:21:31.735176 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:31.735139    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a12c35/payload-processing-external-model-e2e-extmod-a-a12c35-664fd8jf5"]
Nov 01 05:21:31.735582 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:21:31.735539    5689 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="ai-tenant-e2e-extmod-a-a12c35/payload-processing-external-model-e2e-extmod-a-a12c35-664fd8jf5" podUID="e5f6593b-d8a7-434e-bd2a-4555aac39916"
Nov 01 05:21:31.928242 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:31.928199    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:21:32.503654 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:32.503621    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6nz2mn"
Nov 01 05:21:32.547213 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:32.547180    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a12c35/payload-processing-external-model-e2e-extmod-a-a12c35-664fd8jf5"
Nov 01 05:21:32.568173 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:32.568118    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a12c35/payload-processing-external-model-e2e-extmod-a-a12c35-664fd8jf5"
Nov 01 05:21:32.673143 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:32.673110    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5f6593b-d8a7-434e-bd2a-4555aac39916-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e5f6593b-d8a7-434e-bd2a-4555aac39916-routing-overlay-volume\") pod \"e5f6593b-d8a7-434e-bd2a-4555aac39916\" (UID: \"e5f6593b-d8a7-434e-bd2a-4555aac39916\") "
Nov 01 05:21:32.673433 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:32.673399    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5f6593b-d8a7-434e-bd2a-4555aac39916-kube-api-access-vrqrc\" (UniqueName: \"kubernetes.io/projected/e5f6593b-d8a7-434e-bd2a-4555aac39916-kube-api-access-vrqrc\") pod \"e5f6593b-d8a7-434e-bd2a-4555aac39916\" (UID: \"e5f6593b-d8a7-434e-bd2a-4555aac39916\") "
Nov 01 05:21:32.673850 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:32.673816    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5f6593b-d8a7-434e-bd2a-4555aac39916-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5f6593b-d8a7-434e-bd2a-4555aac39916-plugins-config-volume\") pod \"e5f6593b-d8a7-434e-bd2a-4555aac39916\" (UID: \"e5f6593b-d8a7-434e-bd2a-4555aac39916\") "
Nov 01 05:21:32.676369 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:32.676346    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5f6593b-d8a7-434e-bd2a-4555aac39916-plugins-config-volume" pod "e5f6593b-d8a7-434e-bd2a-4555aac39916" (UID: "e5f6593b-d8a7-434e-bd2a-4555aac39916"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:21:32.676912 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:32.676891    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5f6593b-d8a7-434e-bd2a-4555aac39916-routing-overlay-volume" pod "e5f6593b-d8a7-434e-bd2a-4555aac39916" (UID: "e5f6593b-d8a7-434e-bd2a-4555aac39916"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:21:32.690708 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:32.690669    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5f6593b-d8a7-434e-bd2a-4555aac39916-kube-api-access-vrqrc" pod "e5f6593b-d8a7-434e-bd2a-4555aac39916" (UID: "e5f6593b-d8a7-434e-bd2a-4555aac39916"). InnerVolumeSpecName "kube-api-access-vrqrc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:21:32.708300 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:32.708258    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:21:32.776507 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:32.776394    5689 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e5f6593b-d8a7-434e-bd2a-4555aac39916-routing-overlay-volume\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:21:32.776507 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:32.776427    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrqrc\" (UniqueName: \"kubernetes.io/projected/e5f6593b-d8a7-434e-bd2a-4555aac39916-kube-api-access-vrqrc\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:21:32.776507 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:32.776441    5689 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5f6593b-d8a7-434e-bd2a-4555aac39916-plugins-config-volume\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:21:33.283619 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:33.283578    5689 patch_prober.go:29] interesting pod/e2e-worker-w4-a48099-openshift-default-577864759-sgdbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 05:21:33.283812 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:33.283632    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-sgdbb" podUID="3654852a-2923-46d4-a244-8a907545ed9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 05:21:33.529075 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:33.528091    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-sgdbb"
Nov 01 05:21:33.529075 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:33.528862    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-sgdbb"
Nov 01 05:21:33.529661 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:33.529644    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-sgdbb"
Nov 01 05:21:33.553477 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:33.553395    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a12c35/payload-processing-external-model-e2e-extmod-a-a12c35-664fd8jf5"
Nov 01 05:21:33.875042 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:33.874962    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a12c35/payload-processing-external-model-e2e-extmod-a-a12c35-664fd8jf5"]
Nov 01 05:21:33.878734 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:33.878711    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-a12c35/payload-processing-external-model-e2e-extmod-a-a12c35-664fd8jf5"]
Nov 01 05:21:33.890389 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:33.890370    5689 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e5f6593b-d8a7-434e-bd2a-4555aac39916-provider-credentials\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:21:34.068295 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:34.068256    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5f6593b-d8a7-434e-bd2a-4555aac39916" path="/var/lib/kubelet/pods/e5f6593b-d8a7-434e-bd2a-4555aac39916/volumes"
Nov 01 05:21:36.045663 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:36.045633    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-3d70ff/distinct-2-w2-3d70ff-kserve-868b775cd4-nhx4f"]
Nov 01 05:21:36.050593 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:36.050574    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-3d70ff/distinct-2-w2-3d70ff-kserve-868b775cd4-nhx4f"
Nov 01 05:21:36.117507 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:36.117475    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c79347d-eb2e-49b2-af27-48ddcf86b7bb-dshm\") pod \"distinct-2-w2-3d70ff-kserve-868b775cd4-nhx4f\" (UID: \"0c79347d-eb2e-49b2-af27-48ddcf86b7bb\") " pod="e2e-models-e2e-worker-w2-3d70ff/distinct-2-w2-3d70ff-kserve-868b775cd4-nhx4f"
Nov 01 05:21:36.117680 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:36.117631    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c79347d-eb2e-49b2-af27-48ddcf86b7bb-home\") pod \"distinct-2-w2-3d70ff-kserve-868b775cd4-nhx4f\" (UID: \"0c79347d-eb2e-49b2-af27-48ddcf86b7bb\") " pod="e2e-models-e2e-worker-w2-3d70ff/distinct-2-w2-3d70ff-kserve-868b775cd4-nhx4f"
Nov 01 05:21:36.117680 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:36.117670    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c79347d-eb2e-49b2-af27-48ddcf86b7bb-tmp-dir\") pod \"distinct-2-w2-3d70ff-kserve-868b775cd4-nhx4f\" (UID: \"0c79347d-eb2e-49b2-af27-48ddcf86b7bb\") " pod="e2e-models-e2e-worker-w2-3d70ff/distinct-2-w2-3d70ff-kserve-868b775cd4-nhx4f"
Nov 01 05:21:36.117802 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:36.117713    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vrbr\" (UniqueName: \"kubernetes.io/projected/0c79347d-eb2e-49b2-af27-48ddcf86b7bb-kube-api-access-5vrbr\") pod \"distinct-2-w2-3d70ff-kserve-868b775cd4-nhx4f\" (UID: \"0c79347d-eb2e-49b2-af27-48ddcf86b7bb\") " pod="e2e-models-e2e-worker-w2-3d70ff/distinct-2-w2-3d70ff-kserve-868b775cd4-nhx4f"
Nov 01 05:21:36.117802 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:36.117736    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c79347d-eb2e-49b2-af27-48ddcf86b7bb-tls-certs\") pod \"distinct-2-w2-3d70ff-kserve-868b775cd4-nhx4f\" (UID: \"0c79347d-eb2e-49b2-af27-48ddcf86b7bb\") " pod="e2e-models-e2e-worker-w2-3d70ff/distinct-2-w2-3d70ff-kserve-868b775cd4-nhx4f"
Nov 01 05:21:36.117802 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:36.117773    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c79347d-eb2e-49b2-af27-48ddcf86b7bb-model-cache\") pod \"distinct-2-w2-3d70ff-kserve-868b775cd4-nhx4f\" (UID: \"0c79347d-eb2e-49b2-af27-48ddcf86b7bb\") " pod="e2e-models-e2e-worker-w2-3d70ff/distinct-2-w2-3d70ff-kserve-868b775cd4-nhx4f"
Nov 01 05:21:36.151811 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:36.151787    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-3d70ff\"/\"distinct-2-w2-3d70ff-kserve-self-signed-certs\""
Nov 01 05:21:36.153702 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:36.153683    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-3d70ff/distinct-2-w2-3d70ff-kserve-868b775cd4-nhx4f"]
Nov 01 05:21:36.172677 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:36.172655    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:21:36.220163 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:36.219516    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c79347d-eb2e-49b2-af27-48ddcf86b7bb-home\") pod \"distinct-2-w2-3d70ff-kserve-868b775cd4-nhx4f\" (UID: \"0c79347d-eb2e-49b2-af27-48ddcf86b7bb\") " pod="e2e-models-e2e-worker-w2-3d70ff/distinct-2-w2-3d70ff-kserve-868b775cd4-nhx4f"
Nov 01 05:21:36.220163 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:36.219567    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c79347d-eb2e-49b2-af27-48ddcf86b7bb-tmp-dir\") pod \"distinct-2-w2-3d70ff-kserve-868b775cd4-nhx4f\" (UID: \"0c79347d-eb2e-49b2-af27-48ddcf86b7bb\") " pod="e2e-models-e2e-worker-w2-3d70ff/distinct-2-w2-3d70ff-kserve-868b775cd4-nhx4f"
Nov 01 05:21:36.220163 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:36.219634    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vrbr\" (UniqueName: \"kubernetes.io/projected/0c79347d-eb2e-49b2-af27-48ddcf86b7bb-kube-api-access-5vrbr\") pod \"distinct-2-w2-3d70ff-kserve-868b775cd4-nhx4f\" (UID: \"0c79347d-eb2e-49b2-af27-48ddcf86b7bb\") " pod="e2e-models-e2e-worker-w2-3d70ff/distinct-2-w2-3d70ff-kserve-868b775cd4-nhx4f"
Nov 01 05:21:36.220163 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:36.219663    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c79347d-eb2e-49b2-af27-48ddcf86b7bb-tls-certs\") pod \"distinct-2-w2-3d70ff-kserve-868b775cd4-nhx4f\" (UID: \"0c79347d-eb2e-49b2-af27-48ddcf86b7bb\") " pod="e2e-models-e2e-worker-w2-3d70ff/distinct-2-w2-3d70ff-kserve-868b775cd4-nhx4f"
Nov 01 05:21:36.220163 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:36.219725    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c79347d-eb2e-49b2-af27-48ddcf86b7bb-model-cache\") pod \"distinct-2-w2-3d70ff-kserve-868b775cd4-nhx4f\" (UID: \"0c79347d-eb2e-49b2-af27-48ddcf86b7bb\") " pod="e2e-models-e2e-worker-w2-3d70ff/distinct-2-w2-3d70ff-kserve-868b775cd4-nhx4f"
Nov 01 05:21:36.220163 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:36.219797    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c79347d-eb2e-49b2-af27-48ddcf86b7bb-dshm\") pod \"distinct-2-w2-3d70ff-kserve-868b775cd4-nhx4f\" (UID: \"0c79347d-eb2e-49b2-af27-48ddcf86b7bb\") " pod="e2e-models-e2e-worker-w2-3d70ff/distinct-2-w2-3d70ff-kserve-868b775cd4-nhx4f"
Nov 01 05:21:36.220741 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:36.220714    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c79347d-eb2e-49b2-af27-48ddcf86b7bb-home\") pod \"distinct-2-w2-3d70ff-kserve-868b775cd4-nhx4f\" (UID: \"0c79347d-eb2e-49b2-af27-48ddcf86b7bb\") " pod="e2e-models-e2e-worker-w2-3d70ff/distinct-2-w2-3d70ff-kserve-868b775cd4-nhx4f"
Nov 01 05:21:36.220953 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:36.220932    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c79347d-eb2e-49b2-af27-48ddcf86b7bb-model-cache\") pod \"distinct-2-w2-3d70ff-kserve-868b775cd4-nhx4f\" (UID: \"0c79347d-eb2e-49b2-af27-48ddcf86b7bb\") " pod="e2e-models-e2e-worker-w2-3d70ff/distinct-2-w2-3d70ff-kserve-868b775cd4-nhx4f"
Nov 01 05:21:36.221720 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:36.221689    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c79347d-eb2e-49b2-af27-48ddcf86b7bb-tmp-dir\") pod \"distinct-2-w2-3d70ff-kserve-868b775cd4-nhx4f\" (UID: \"0c79347d-eb2e-49b2-af27-48ddcf86b7bb\") " pod="e2e-models-e2e-worker-w2-3d70ff/distinct-2-w2-3d70ff-kserve-868b775cd4-nhx4f"
Nov 01 05:21:36.223382 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:36.223354    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c79347d-eb2e-49b2-af27-48ddcf86b7bb-dshm\") pod \"distinct-2-w2-3d70ff-kserve-868b775cd4-nhx4f\" (UID: \"0c79347d-eb2e-49b2-af27-48ddcf86b7bb\") " pod="e2e-models-e2e-worker-w2-3d70ff/distinct-2-w2-3d70ff-kserve-868b775cd4-nhx4f"
Nov 01 05:21:36.224560 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:36.224535    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c79347d-eb2e-49b2-af27-48ddcf86b7bb-tls-certs\") pod \"distinct-2-w2-3d70ff-kserve-868b775cd4-nhx4f\" (UID: \"0c79347d-eb2e-49b2-af27-48ddcf86b7bb\") " pod="e2e-models-e2e-worker-w2-3d70ff/distinct-2-w2-3d70ff-kserve-868b775cd4-nhx4f"
Nov 01 05:21:36.285935 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:36.285911    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vrbr\" (UniqueName: \"kubernetes.io/projected/0c79347d-eb2e-49b2-af27-48ddcf86b7bb-kube-api-access-5vrbr\") pod \"distinct-2-w2-3d70ff-kserve-868b775cd4-nhx4f\" (UID: \"0c79347d-eb2e-49b2-af27-48ddcf86b7bb\") " pod="e2e-models-e2e-worker-w2-3d70ff/distinct-2-w2-3d70ff-kserve-868b775cd4-nhx4f"
Nov 01 05:21:36.362273 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:36.362200    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-3d70ff/distinct-2-w2-3d70ff-kserve-868b775cd4-nhx4f"
Nov 01 05:21:36.776598 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:36.776563    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-3d70ff/distinct-2-w2-3d70ff-kserve-868b775cd4-nhx4f"]
Nov 01 05:21:37.520094 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:37.503332    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:21:37.575202 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:37.575163    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-3d70ff/distinct-2-w2-3d70ff-kserve-868b775cd4-nhx4f" event={"ID":"0c79347d-eb2e-49b2-af27-48ddcf86b7bb","Type":"ContainerStarted","Data":"43ab0c1f2c3c4e1ea9d4f3b52217df17a7c8d5d9aab7c8b3959f70b2c440c412"}
Nov 01 05:21:37.575202 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:37.575208    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-3d70ff/distinct-2-w2-3d70ff-kserve-868b775cd4-nhx4f" event={"ID":"0c79347d-eb2e-49b2-af27-48ddcf86b7bb","Type":"ContainerStarted","Data":"90fb9e6596293c6db025197add3e4485e6b17ee8953739f608390cbef3c50339"}
Nov 01 05:21:37.691873 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:37.691800    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-3d70ff/distinct-2-w2-3d70ff-kserve-868b775cd4-nhx4f" podStartSLOduration=1.691781965 podStartE2EDuration="1.691781965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:21:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:21:37.691191923 +0000 UTC m=+1075.761988639" watchObservedRunningTime="2026-11-01 05:21:37.691781965 +0000 UTC m=+1075.762578687"
Nov 01 05:21:38.362560 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:38.362520    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-3d70ff/distinct-2-w2-3d70ff-kserve-868b775cd4-nhx4f"
Nov 01 05:21:39.060143 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:39.060112    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-kf57z"]
Nov 01 05:21:39.064495 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:39.064466    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-kf57z"
Nov 01 05:21:39.168384 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:39.168348    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb42038d-b688-41da-9700-955bbda47478-workload-socket\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-kf57z\" (UID: \"eb42038d-b688-41da-9700-955bbda47478\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-kf57z"
Nov 01 05:21:39.168565 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:39.168394    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb42038d-b688-41da-9700-955bbda47478-istio-envoy\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-kf57z\" (UID: \"eb42038d-b688-41da-9700-955bbda47478\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-kf57z"
Nov 01 05:21:39.168565 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:39.168437    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9rhw\" (UniqueName: \"kubernetes.io/projected/eb42038d-b688-41da-9700-955bbda47478-kube-api-access-v9rhw\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-kf57z\" (UID: \"eb42038d-b688-41da-9700-955bbda47478\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-kf57z"
Nov 01 05:21:39.168565 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:39.168481    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-kf57z"]
Nov 01 05:21:39.168764 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:39.168598    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb42038d-b688-41da-9700-955bbda47478-istiod-ca-cert\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-kf57z\" (UID: \"eb42038d-b688-41da-9700-955bbda47478\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-kf57z"
Nov 01 05:21:39.168764 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:39.168648    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb42038d-b688-41da-9700-955bbda47478-istio-token\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-kf57z\" (UID: \"eb42038d-b688-41da-9700-955bbda47478\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-kf57z"
Nov 01 05:21:39.168764 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:39.168715    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb42038d-b688-41da-9700-955bbda47478-istio-podinfo\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-kf57z\" (UID: \"eb42038d-b688-41da-9700-955bbda47478\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-kf57z"
Nov 01 05:21:39.168764 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:39.168754    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb42038d-b688-41da-9700-955bbda47478-workload-certs\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-kf57z\" (UID: \"eb42038d-b688-41da-9700-955bbda47478\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-kf57z"
Nov 01 05:21:39.168974 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:39.168932    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb42038d-b688-41da-9700-955bbda47478-istio-data\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-kf57z\" (UID: \"eb42038d-b688-41da-9700-955bbda47478\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-kf57z"
Nov 01 05:21:39.169030 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:39.168967    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb42038d-b688-41da-9700-955bbda47478-credential-socket\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-kf57z\" (UID: \"eb42038d-b688-41da-9700-955bbda47478\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-kf57z"
Nov 01 05:21:39.270670 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:39.270631    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb42038d-b688-41da-9700-955bbda47478-workload-socket\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-kf57z\" (UID: \"eb42038d-b688-41da-9700-955bbda47478\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-kf57z"
Nov 01 05:21:39.270670 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:39.270673    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb42038d-b688-41da-9700-955bbda47478-istio-envoy\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-kf57z\" (UID: \"eb42038d-b688-41da-9700-955bbda47478\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-kf57z"
Nov 01 05:21:39.270876 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:39.270699    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9rhw\" (UniqueName: \"kubernetes.io/projected/eb42038d-b688-41da-9700-955bbda47478-kube-api-access-v9rhw\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-kf57z\" (UID: \"eb42038d-b688-41da-9700-955bbda47478\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-kf57z"
Nov 01 05:21:39.270876 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:39.270740    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb42038d-b688-41da-9700-955bbda47478-istiod-ca-cert\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-kf57z\" (UID: \"eb42038d-b688-41da-9700-955bbda47478\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-kf57z"
Nov 01 05:21:39.270876 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:39.270764    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb42038d-b688-41da-9700-955bbda47478-istio-token\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-kf57z\" (UID: \"eb42038d-b688-41da-9700-955bbda47478\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-kf57z"
Nov 01 05:21:39.270876 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:39.270793    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb42038d-b688-41da-9700-955bbda47478-istio-podinfo\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-kf57z\" (UID: \"eb42038d-b688-41da-9700-955bbda47478\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-kf57z"
Nov 01 05:21:39.270876 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:39.270848    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb42038d-b688-41da-9700-955bbda47478-workload-certs\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-kf57z\" (UID: \"eb42038d-b688-41da-9700-955bbda47478\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-kf57z"
Nov 01 05:21:39.271122 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:39.270908    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb42038d-b688-41da-9700-955bbda47478-istio-data\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-kf57z\" (UID: \"eb42038d-b688-41da-9700-955bbda47478\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-kf57z"
Nov 01 05:21:39.271122 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:39.270938    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb42038d-b688-41da-9700-955bbda47478-credential-socket\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-kf57z\" (UID: \"eb42038d-b688-41da-9700-955bbda47478\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-kf57z"
Nov 01 05:21:39.271122 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:39.271075    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb42038d-b688-41da-9700-955bbda47478-workload-socket\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-kf57z\" (UID: \"eb42038d-b688-41da-9700-955bbda47478\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-kf57z"
Nov 01 05:21:39.271321 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:39.271294    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb42038d-b688-41da-9700-955bbda47478-workload-certs\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-kf57z\" (UID: \"eb42038d-b688-41da-9700-955bbda47478\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-kf57z"
Nov 01 05:21:39.271648 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:39.271357    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb42038d-b688-41da-9700-955bbda47478-credential-socket\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-kf57z\" (UID: \"eb42038d-b688-41da-9700-955bbda47478\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-kf57z"
Nov 01 05:21:39.271648 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:39.271420    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb42038d-b688-41da-9700-955bbda47478-istio-data\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-kf57z\" (UID: \"eb42038d-b688-41da-9700-955bbda47478\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-kf57z"
Nov 01 05:21:39.271648 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:39.271528    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb42038d-b688-41da-9700-955bbda47478-istiod-ca-cert\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-kf57z\" (UID: \"eb42038d-b688-41da-9700-955bbda47478\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-kf57z"
Nov 01 05:21:39.273107 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:39.273081    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb42038d-b688-41da-9700-955bbda47478-istio-envoy\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-kf57z\" (UID: \"eb42038d-b688-41da-9700-955bbda47478\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-kf57z"
Nov 01 05:21:39.273432 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:39.273410    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb42038d-b688-41da-9700-955bbda47478-istio-podinfo\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-kf57z\" (UID: \"eb42038d-b688-41da-9700-955bbda47478\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-kf57z"
Nov 01 05:21:39.331157 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:39.331098    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb42038d-b688-41da-9700-955bbda47478-istio-token\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-kf57z\" (UID: \"eb42038d-b688-41da-9700-955bbda47478\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-kf57z"
Nov 01 05:21:39.331255 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:39.331234    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9rhw\" (UniqueName: \"kubernetes.io/projected/eb42038d-b688-41da-9700-955bbda47478-kube-api-access-v9rhw\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-kf57z\" (UID: \"eb42038d-b688-41da-9700-955bbda47478\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-kf57z"
Nov 01 05:21:39.378233 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:39.378211    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-kf57z"
Nov 01 05:21:39.639157 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:39.639078    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:21:39.812945 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:39.812921    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-kf57z"]
Nov 01 05:21:39.815425 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:21:39.815399    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb42038d_b688_41da_9700_955bbda47478.slice/crio-1360db150d11086e20a0c3adb6a63ea6c7c0b4f8b2e7ff46eb8b63031f131d49 WatchSource:0}: Error finding container 1360db150d11086e20a0c3adb6a63ea6c7c0b4f8b2e7ff46eb8b63031f131d49: Status 404 returned error can't find the container with id 1360db150d11086e20a0c3adb6a63ea6c7c0b4f8b2e7ff46eb8b63031f131d49
Nov 01 05:21:39.817852 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:39.817818    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:21:39.817987 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:39.817895    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:21:39.817987 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:39.817937    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:21:40.601634 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:40.601593    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:21:40.611321 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:40.611289    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-kf57z" event={"ID":"eb42038d-b688-41da-9700-955bbda47478","Type":"ContainerStarted","Data":"ed565d2179bb0eee7829320de6b23f4a7c3732ad2ab158f4075a8a99d932a6c5"}
Nov 01 05:21:40.611507 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:40.611378    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-kf57z" event={"ID":"eb42038d-b688-41da-9700-955bbda47478","Type":"ContainerStarted","Data":"1360db150d11086e20a0c3adb6a63ea6c7c0b4f8b2e7ff46eb8b63031f131d49"}
Nov 01 05:21:40.769741 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:40.769688    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-kf57z" podStartSLOduration=1.7696749010000001 podStartE2EDuration="1.769674901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:21:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:21:40.767961649 +0000 UTC m=+1078.838758369" watchObservedRunningTime="2026-11-01 05:21:40.769674901 +0000 UTC m=+1078.840471621"
Nov 01 05:21:41.118739 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:41.118699    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd"]
Nov 01 05:21:41.123337 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:41.123316    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd"
Nov 01 05:21:41.178498 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:41.176607    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-5451cb-openshift-default-dockercfg-cck8m\""
Nov 01 05:21:41.239133 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:41.238541    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd"]
Nov 01 05:21:41.242161 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:41.239316    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50-istio-token\") pod \"e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd\" (UID: \"a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50\") " pod="openshift-ingress/e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd"
Nov 01 05:21:41.242161 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:41.239368    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50-workload-certs\") pod \"e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd\" (UID: \"a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50\") " pod="openshift-ingress/e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd"
Nov 01 05:21:41.242161 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:41.239488    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50-istio-data\") pod \"e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd\" (UID: \"a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50\") " pod="openshift-ingress/e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd"
Nov 01 05:21:41.242161 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:41.239519    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdm75\" (UniqueName: \"kubernetes.io/projected/a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50-kube-api-access-zdm75\") pod \"e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd\" (UID: \"a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50\") " pod="openshift-ingress/e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd"
Nov 01 05:21:41.242161 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:41.239562    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50-istiod-ca-cert\") pod \"e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd\" (UID: \"a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50\") " pod="openshift-ingress/e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd"
Nov 01 05:21:41.242161 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:41.239625    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50-istio-envoy\") pod \"e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd\" (UID: \"a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50\") " pod="openshift-ingress/e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd"
Nov 01 05:21:41.242161 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:41.239716    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50-credential-socket\") pod \"e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd\" (UID: \"a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50\") " pod="openshift-ingress/e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd"
Nov 01 05:21:41.242161 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:41.239780    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50-workload-socket\") pod \"e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd\" (UID: \"a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50\") " pod="openshift-ingress/e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd"
Nov 01 05:21:41.242161 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:41.239804    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50-istio-podinfo\") pod \"e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd\" (UID: \"a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50\") " pod="openshift-ingress/e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd"
Nov 01 05:21:41.367475 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:41.353690    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50-istio-data\") pod \"e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd\" (UID: \"a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50\") " pod="openshift-ingress/e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd"
Nov 01 05:21:41.367475 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:41.353736    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdm75\" (UniqueName: \"kubernetes.io/projected/a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50-kube-api-access-zdm75\") pod \"e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd\" (UID: \"a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50\") " pod="openshift-ingress/e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd"
Nov 01 05:21:41.367475 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:41.353783    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50-istiod-ca-cert\") pod \"e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd\" (UID: \"a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50\") " pod="openshift-ingress/e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd"
Nov 01 05:21:41.367475 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:41.353848    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50-istio-envoy\") pod \"e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd\" (UID: \"a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50\") " pod="openshift-ingress/e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd"
Nov 01 05:21:41.367475 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:41.353932    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50-credential-socket\") pod \"e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd\" (UID: \"a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50\") " pod="openshift-ingress/e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd"
Nov 01 05:21:41.367475 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:41.353995    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50-workload-socket\") pod \"e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd\" (UID: \"a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50\") " pod="openshift-ingress/e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd"
Nov 01 05:21:41.367475 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:41.354017    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50-istio-podinfo\") pod \"e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd\" (UID: \"a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50\") " pod="openshift-ingress/e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd"
Nov 01 05:21:41.367475 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:41.354081    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50-istio-token\") pod \"e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd\" (UID: \"a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50\") " pod="openshift-ingress/e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd"
Nov 01 05:21:41.367475 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:41.354114    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50-workload-certs\") pod \"e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd\" (UID: \"a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50\") " pod="openshift-ingress/e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd"
Nov 01 05:21:41.367475 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:41.354608    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50-workload-certs\") pod \"e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd\" (UID: \"a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50\") " pod="openshift-ingress/e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd"
Nov 01 05:21:41.367475 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:41.354807    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50-istio-data\") pod \"e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd\" (UID: \"a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50\") " pod="openshift-ingress/e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd"
Nov 01 05:21:41.367475 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:41.355735    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50-istiod-ca-cert\") pod \"e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd\" (UID: \"a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50\") " pod="openshift-ingress/e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd"
Nov 01 05:21:41.367475 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:41.358689    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50-workload-socket\") pod \"e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd\" (UID: \"a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50\") " pod="openshift-ingress/e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd"
Nov 01 05:21:41.367475 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:41.359346    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50-credential-socket\") pod \"e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd\" (UID: \"a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50\") " pod="openshift-ingress/e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd"
Nov 01 05:21:41.407531 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:41.371817    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:21:41.407531 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:41.378781    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-kf57z"
Nov 01 05:21:41.407531 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:41.387547    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50-istio-envoy\") pod \"e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd\" (UID: \"a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50\") " pod="openshift-ingress/e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd"
Nov 01 05:21:41.413515 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:41.408642    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50-istio-podinfo\") pod \"e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd\" (UID: \"a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50\") " pod="openshift-ingress/e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd"
Nov 01 05:21:41.413515 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:41.408686    5689 patch_prober.go:29] interesting pod/e2e-worker-w4-a48099-openshift-default-577864759-kf57z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 01 05:21:41.413515 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:41.408737    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-kf57z" podUID="eb42038d-b688-41da-9700-955bbda47478" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 01 05:21:41.415609 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:41.415579    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdm75\" (UniqueName: \"kubernetes.io/projected/a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50-kube-api-access-zdm75\") pod \"e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd\" (UID: \"a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50\") " pod="openshift-ingress/e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd"
Nov 01 05:21:41.420071 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:41.420045    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50-istio-token\") pod \"e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd\" (UID: \"a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50\") " pod="openshift-ingress/e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd"
Nov 01 05:21:41.437179 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:41.437156    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd"
Nov 01 05:21:41.643972 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:41.642815    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:21:41.717391 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:41.717308    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2864b6-d5585ff4c-x2hfw"]
Nov 01 05:21:41.893478 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:41.893155    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd"]
Nov 01 05:21:41.902478 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:21:41.902424    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda43ac5f8_e7ab_4e5d_a8a9_129926f1ba50.slice/crio-16768705388d8a6a3b95f6b52888c22ad3712b4f0858c8799f14c6c554b19bf0 WatchSource:0}: Error finding container 16768705388d8a6a3b95f6b52888c22ad3712b4f0858c8799f14c6c554b19bf0: Status 404 returned error can't find the container with id 16768705388d8a6a3b95f6b52888c22ad3712b4f0858c8799f14c6c554b19bf0
Nov 01 05:21:41.906015 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:41.905984    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:21:41.906116 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:41.906062    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:21:41.906177 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:41.906131    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:21:42.009465 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:42.009428    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:21:42.200481 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:42.196386    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:21:42.383479 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:42.382775    5689 patch_prober.go:29] interesting pod/e2e-worker-w4-a48099-openshift-default-577864759-kf57z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 01 05:21:42.383479 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:42.382836    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-kf57z" podUID="eb42038d-b688-41da-9700-955bbda47478" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 01 05:21:42.630117 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:42.630027    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd" event={"ID":"a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50","Type":"ContainerStarted","Data":"e471ff50c9e2245d141f675eba5be657e2eab2385dfd2d032238ae783a61319e"}
Nov 01 05:21:42.630117 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:42.630092    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd" event={"ID":"a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50","Type":"ContainerStarted","Data":"16768705388d8a6a3b95f6b52888c22ad3712b4f0858c8799f14c6c554b19bf0"}
Nov 01 05:21:42.763777 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:42.763728    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd" podStartSLOduration=1.7637152889999999 podStartE2EDuration="1.763715289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:21:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:21:42.76183327 +0000 UTC m=+1080.832629991" watchObservedRunningTime="2026-11-01 05:21:42.763715289 +0000 UTC m=+1080.834512009"
Nov 01 05:21:42.801049 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:42.801011    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:21:43.380478 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:43.379895    5689 patch_prober.go:29] interesting pod/e2e-worker-w4-a48099-openshift-default-577864759-kf57z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 01 05:21:43.380478 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:43.379976    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-kf57z" podUID="eb42038d-b688-41da-9700-955bbda47478" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 01 05:21:43.446475 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:43.444622    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd"
Nov 01 05:21:43.465437 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:43.464947    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd"
Nov 01 05:21:43.635238 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:43.635163    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd"
Nov 01 05:21:43.636192 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:43.636169    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd"
Nov 01 05:21:44.388858 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:44.388822    5689 patch_prober.go:29] interesting pod/e2e-worker-w4-a48099-openshift-default-577864759-kf57z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 01 05:21:44.390018 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:44.389986    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-kf57z" podUID="eb42038d-b688-41da-9700-955bbda47478" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 01 05:21:45.385487 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:45.379730    5689 patch_prober.go:29] interesting pod/e2e-worker-w4-a48099-openshift-default-577864759-kf57z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 01 05:21:45.385487 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:45.379796    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-kf57z" podUID="eb42038d-b688-41da-9700-955bbda47478" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 01 05:21:46.363253 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:46.363219    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-3d70ff/distinct-2-w2-3d70ff-kserve-868b775cd4-nhx4f"
Nov 01 05:21:46.375695 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:46.375670    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-3d70ff/distinct-2-w2-3d70ff-kserve-868b775cd4-nhx4f"
Nov 01 05:21:46.379022 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:46.378991    5689 patch_prober.go:29] interesting pod/e2e-worker-w4-a48099-openshift-default-577864759-kf57z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 01 05:21:46.379152 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:46.379040    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-kf57z" podUID="eb42038d-b688-41da-9700-955bbda47478" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 01 05:21:46.462242 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:46.462212    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:21:46.723479 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:46.722938    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2864b6-d5585ff4c-x2hfw" podUID="adbbc4d7-e023-4063-be79-7da0aa65cd03" containerName="payload-pre-processing" containerID="cri-o://dc03bd09ba0c0466c231814e97da0a917631ec77c183d4b1f40bf9312ca36f7a" gracePeriod=25
Nov 01 05:21:46.733522 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:46.724717    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-3d70ff/distinct-2-w2-3d70ff-kserve-868b775cd4-nhx4f"
Nov 01 05:21:47.123678 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:47.123466    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:21:47.168377 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:47.168353    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2864b6-d5585ff4c-x2hfw"
Nov 01 05:21:47.198499 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:47.198062    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adbbc4d7-e023-4063-be79-7da0aa65cd03-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adbbc4d7-e023-4063-be79-7da0aa65cd03-plugins-config-volume\") pod \"adbbc4d7-e023-4063-be79-7da0aa65cd03\" (UID: \"adbbc4d7-e023-4063-be79-7da0aa65cd03\") "
Nov 01 05:21:47.198499 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:47.198214    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adbbc4d7-e023-4063-be79-7da0aa65cd03-kube-api-access-g982j\" (UniqueName: \"kubernetes.io/projected/adbbc4d7-e023-4063-be79-7da0aa65cd03-kube-api-access-g982j\") pod \"adbbc4d7-e023-4063-be79-7da0aa65cd03\" (UID: \"adbbc4d7-e023-4063-be79-7da0aa65cd03\") "
Nov 01 05:21:47.199382 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:47.199356    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adbbc4d7-e023-4063-be79-7da0aa65cd03-plugins-config-volume" pod "adbbc4d7-e023-4063-be79-7da0aa65cd03" (UID: "adbbc4d7-e023-4063-be79-7da0aa65cd03"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:21:47.208437 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:47.208401    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adbbc4d7-e023-4063-be79-7da0aa65cd03-kube-api-access-g982j" pod "adbbc4d7-e023-4063-be79-7da0aa65cd03" (UID: "adbbc4d7-e023-4063-be79-7da0aa65cd03"). InnerVolumeSpecName "kube-api-access-g982j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:21:47.299921 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:47.299884    5689 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adbbc4d7-e023-4063-be79-7da0aa65cd03-plugins-config-volume\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:21:47.299921 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:47.299925    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g982j\" (UniqueName: \"kubernetes.io/projected/adbbc4d7-e023-4063-be79-7da0aa65cd03-kube-api-access-g982j\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:21:47.316048 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:47.316006    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:21:47.497063 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:47.496681    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:21:47.669945 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:47.669905    5689 generic.go:356] "Generic (PLEG): container finished" podID="adbbc4d7-e023-4063-be79-7da0aa65cd03" containerID="dc03bd09ba0c0466c231814e97da0a917631ec77c183d4b1f40bf9312ca36f7a" exitCode=0
Nov 01 05:21:47.670743 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:47.670537    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2864b6-d5585ff4c-x2hfw" event={"ID":"adbbc4d7-e023-4063-be79-7da0aa65cd03","Type":"ContainerDied","Data":"dc03bd09ba0c0466c231814e97da0a917631ec77c183d4b1f40bf9312ca36f7a"}
Nov 01 05:21:47.670743 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:47.670564    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2864b6-d5585ff4c-x2hfw"
Nov 01 05:21:47.670743 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:47.670591    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2864b6-d5585ff4c-x2hfw" event={"ID":"adbbc4d7-e023-4063-be79-7da0aa65cd03","Type":"ContainerDied","Data":"51e58bdfc26061adf91e674420698a00f0b270f8b8092a12267831e0c015e557"}
Nov 01 05:21:47.670743 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:47.670623    5689 scope.go:122] "RemoveContainer" containerID="dc03bd09ba0c0466c231814e97da0a917631ec77c183d4b1f40bf9312ca36f7a"
Nov 01 05:21:47.698153 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:47.698051    5689 scope.go:122] "RemoveContainer" containerID="dc03bd09ba0c0466c231814e97da0a917631ec77c183d4b1f40bf9312ca36f7a"
Nov 01 05:21:47.698442 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:21:47.698412    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc03bd09ba0c0466c231814e97da0a917631ec77c183d4b1f40bf9312ca36f7a\": container with ID starting with dc03bd09ba0c0466c231814e97da0a917631ec77c183d4b1f40bf9312ca36f7a not found: ID does not exist" containerID="dc03bd09ba0c0466c231814e97da0a917631ec77c183d4b1f40bf9312ca36f7a"
Nov 01 05:21:47.698545 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:47.698473    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc03bd09ba0c0466c231814e97da0a917631ec77c183d4b1f40bf9312ca36f7a"} err="failed to get container status \"dc03bd09ba0c0466c231814e97da0a917631ec77c183d4b1f40bf9312ca36f7a\": rpc error: code = NotFound desc = could not find container \"dc03bd09ba0c0466c231814e97da0a917631ec77c183d4b1f40bf9312ca36f7a\": container with ID starting with dc03bd09ba0c0466c231814e97da0a917631ec77c183d4b1f40bf9312ca36f7a not found: ID does not exist"
Nov 01 05:21:47.853107 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:47.847414    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2864b6-d5585ff4c-x2hfw"]
Nov 01 05:21:47.854492 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:47.854444    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2864b6-d5585ff4c-x2hfw"]
Nov 01 05:21:48.021788 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:48.008003    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:21:48.104527 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:48.103214    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adbbc4d7-e023-4063-be79-7da0aa65cd03" path="/var/lib/kubelet/pods/adbbc4d7-e023-4063-be79-7da0aa65cd03/volumes"
Nov 01 05:21:48.429479 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:48.428434    5689 patch_prober.go:29] interesting pod/e2e-worker-w4-a48099-openshift-default-577864759-kf57z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 05:21:48.429479 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:48.428516    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-kf57z" podUID="eb42038d-b688-41da-9700-955bbda47478" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 05:21:48.980554 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:48.980526    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-kf57z"
Nov 01 05:21:48.981004 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:48.980821    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-kf57z"
Nov 01 05:21:48.981563 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:48.981548    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-kf57z"
Nov 01 05:21:50.407092 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:50.407017    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:21:50.590363 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:50.590331    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:21:51.115611 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:51.115576    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5451cb-794c5878c5-4mnnq"]
Nov 01 05:21:51.116091 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:51.116078    5689 cpu_manager.go:414] "containerMap: removing container" podUID="adbbc4d7-e023-4063-be79-7da0aa65cd03" containerName="payload-pre-processing"
Nov 01 05:21:51.116146 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:51.116092    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adbbc4d7-e023-4063-be79-7da0aa65cd03" containerName="payload-pre-processing"
Nov 01 05:21:51.116189 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:51.116182    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="adbbc4d7-e023-4063-be79-7da0aa65cd03" containerName="payload-pre-processing"
Nov 01 05:21:51.120598 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:51.120581    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5451cb-794c5878c5-4mnnq"
Nov 01 05:21:51.195182 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:51.195155    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61f28ebd-0530-4058-9f4b-a3b17b6c04aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5451cb-794c5878c5-4mnnq\" (UID: \"61f28ebd-0530-4058-9f4b-a3b17b6c04aa\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5451cb-794c5878c5-4mnnq"
Nov 01 05:21:51.195318 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:51.195281    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k57mn\" (UniqueName: \"kubernetes.io/projected/61f28ebd-0530-4058-9f4b-a3b17b6c04aa-kube-api-access-k57mn\") pod \"payload-pre-processing-e2e-ipp-b-5451cb-794c5878c5-4mnnq\" (UID: \"61f28ebd-0530-4058-9f4b-a3b17b6c04aa\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5451cb-794c5878c5-4mnnq"
Nov 01 05:21:51.222117 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:51.222090    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-5451cb\""
Nov 01 05:21:51.222117 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:51.222105    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-5451cb-dockercfg-l5swb\""
Nov 01 05:21:51.224212 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:51.224190    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5451cb-794c5878c5-4mnnq"]
Nov 01 05:21:51.296848 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:51.296813    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61f28ebd-0530-4058-9f4b-a3b17b6c04aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5451cb-794c5878c5-4mnnq\" (UID: \"61f28ebd-0530-4058-9f4b-a3b17b6c04aa\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5451cb-794c5878c5-4mnnq"
Nov 01 05:21:51.296975 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:51.296922    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k57mn\" (UniqueName: \"kubernetes.io/projected/61f28ebd-0530-4058-9f4b-a3b17b6c04aa-kube-api-access-k57mn\") pod \"payload-pre-processing-e2e-ipp-b-5451cb-794c5878c5-4mnnq\" (UID: \"61f28ebd-0530-4058-9f4b-a3b17b6c04aa\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5451cb-794c5878c5-4mnnq"
Nov 01 05:21:51.297630 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:51.297613    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61f28ebd-0530-4058-9f4b-a3b17b6c04aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5451cb-794c5878c5-4mnnq\" (UID: \"61f28ebd-0530-4058-9f4b-a3b17b6c04aa\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5451cb-794c5878c5-4mnnq"
Nov 01 05:21:51.357764 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:51.357742    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k57mn\" (UniqueName: \"kubernetes.io/projected/61f28ebd-0530-4058-9f4b-a3b17b6c04aa-kube-api-access-k57mn\") pod \"payload-pre-processing-e2e-ipp-b-5451cb-794c5878c5-4mnnq\" (UID: \"61f28ebd-0530-4058-9f4b-a3b17b6c04aa\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5451cb-794c5878c5-4mnnq"
Nov 01 05:21:51.433773 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:51.433746    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5451cb-794c5878c5-4mnnq"
Nov 01 05:21:51.851070 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:51.851044    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5451cb-794c5878c5-4mnnq"]
Nov 01 05:21:51.853959 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:21:51.853935    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61f28ebd_0530_4058_9f4b_a3b17b6c04aa.slice/crio-c548a59e6b4d63415ebe4d6f900ed88f5dc78b6a2367e78b4017c0d81f8fe8c5 WatchSource:0}: Error finding container c548a59e6b4d63415ebe4d6f900ed88f5dc78b6a2367e78b4017c0d81f8fe8c5: Status 404 returned error can't find the container with id c548a59e6b4d63415ebe4d6f900ed88f5dc78b6a2367e78b4017c0d81f8fe8c5
Nov 01 05:21:52.732564 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:52.732511    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5451cb-794c5878c5-4mnnq" event={"ID":"61f28ebd-0530-4058-9f4b-a3b17b6c04aa","Type":"ContainerStarted","Data":"60630d5c73fd01c144f293e047ea79a077be65de9c9b40a0fb95d5dc3c18ada9"}
Nov 01 05:21:52.733038 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:52.732622    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5451cb-794c5878c5-4mnnq" event={"ID":"61f28ebd-0530-4058-9f4b-a3b17b6c04aa","Type":"ContainerStarted","Data":"c548a59e6b4d63415ebe4d6f900ed88f5dc78b6a2367e78b4017c0d81f8fe8c5"}
Nov 01 05:21:52.847921 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:52.847872    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5451cb-794c5878c5-4mnnq" podStartSLOduration=1.8478606210000001 podStartE2EDuration="1.847860621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:21:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:21:52.847344344 +0000 UTC m=+1090.918141062" watchObservedRunningTime="2026-11-01 05:21:52.847860621 +0000 UTC m=+1090.918657341"
Nov 01 05:21:53.137534 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:53.128227    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:21:53.738262 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:53.738236    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5451cb-794c5878c5-4mnnq"
Nov 01 05:21:57.852706 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:57.852633    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:21:59.625504 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:21:59.625359    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:22:00.069192 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:22:00.069158    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:22:00.244219 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:00.244178    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:22:02.609029 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:02.608997    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:22:04.745808 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:04.745779    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5451cb-794c5878c5-4mnnq"
Nov 01 05:22:07.337698 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:07.337592    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:22:10.347953 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:10.347922    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr"]
Nov 01 05:22:10.351964 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:10.351947    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr"
Nov 01 05:22:10.430106 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:10.430075    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54adae87-71b3-4dfa-8eb0-3ad86c47e5a8-istio-token\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr\" (UID: \"54adae87-71b3-4dfa-8eb0-3ad86c47e5a8\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr"
Nov 01 05:22:10.430106 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:10.430110    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54adae87-71b3-4dfa-8eb0-3ad86c47e5a8-istiod-ca-cert\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr\" (UID: \"54adae87-71b3-4dfa-8eb0-3ad86c47e5a8\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr"
Nov 01 05:22:10.430268 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:10.430135    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54adae87-71b3-4dfa-8eb0-3ad86c47e5a8-istio-podinfo\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr\" (UID: \"54adae87-71b3-4dfa-8eb0-3ad86c47e5a8\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr"
Nov 01 05:22:10.430319 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:10.430270    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54adae87-71b3-4dfa-8eb0-3ad86c47e5a8-istio-envoy\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr\" (UID: \"54adae87-71b3-4dfa-8eb0-3ad86c47e5a8\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr"
Nov 01 05:22:10.430372 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:10.430325    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54adae87-71b3-4dfa-8eb0-3ad86c47e5a8-workload-certs\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr\" (UID: \"54adae87-71b3-4dfa-8eb0-3ad86c47e5a8\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr"
Nov 01 05:22:10.430415 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:10.430385    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54adae87-71b3-4dfa-8eb0-3ad86c47e5a8-workload-socket\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr\" (UID: \"54adae87-71b3-4dfa-8eb0-3ad86c47e5a8\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr"
Nov 01 05:22:10.430484 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:10.430437    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54adae87-71b3-4dfa-8eb0-3ad86c47e5a8-istio-data\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr\" (UID: \"54adae87-71b3-4dfa-8eb0-3ad86c47e5a8\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr"
Nov 01 05:22:10.430556 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:10.430532    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzsvx\" (UniqueName: \"kubernetes.io/projected/54adae87-71b3-4dfa-8eb0-3ad86c47e5a8-kube-api-access-mzsvx\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr\" (UID: \"54adae87-71b3-4dfa-8eb0-3ad86c47e5a8\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr"
Nov 01 05:22:10.430611 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:10.430587    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54adae87-71b3-4dfa-8eb0-3ad86c47e5a8-credential-socket\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr\" (UID: \"54adae87-71b3-4dfa-8eb0-3ad86c47e5a8\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr"
Nov 01 05:22:10.456070 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:10.456043    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr"]
Nov 01 05:22:10.531970 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:10.531940    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54adae87-71b3-4dfa-8eb0-3ad86c47e5a8-istio-podinfo\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr\" (UID: \"54adae87-71b3-4dfa-8eb0-3ad86c47e5a8\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr"
Nov 01 05:22:10.532137 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:10.531985    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54adae87-71b3-4dfa-8eb0-3ad86c47e5a8-istio-envoy\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr\" (UID: \"54adae87-71b3-4dfa-8eb0-3ad86c47e5a8\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr"
Nov 01 05:22:10.532137 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:10.532029    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54adae87-71b3-4dfa-8eb0-3ad86c47e5a8-workload-certs\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr\" (UID: \"54adae87-71b3-4dfa-8eb0-3ad86c47e5a8\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr"
Nov 01 05:22:10.532239 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:10.532173    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54adae87-71b3-4dfa-8eb0-3ad86c47e5a8-workload-socket\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr\" (UID: \"54adae87-71b3-4dfa-8eb0-3ad86c47e5a8\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr"
Nov 01 05:22:10.532300 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:10.532276    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54adae87-71b3-4dfa-8eb0-3ad86c47e5a8-istio-data\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr\" (UID: \"54adae87-71b3-4dfa-8eb0-3ad86c47e5a8\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr"
Nov 01 05:22:10.532356 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:10.532315    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzsvx\" (UniqueName: \"kubernetes.io/projected/54adae87-71b3-4dfa-8eb0-3ad86c47e5a8-kube-api-access-mzsvx\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr\" (UID: \"54adae87-71b3-4dfa-8eb0-3ad86c47e5a8\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr"
Nov 01 05:22:10.532411 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:10.532353    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54adae87-71b3-4dfa-8eb0-3ad86c47e5a8-credential-socket\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr\" (UID: \"54adae87-71b3-4dfa-8eb0-3ad86c47e5a8\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr"
Nov 01 05:22:10.532507 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:10.532480    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54adae87-71b3-4dfa-8eb0-3ad86c47e5a8-istio-token\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr\" (UID: \"54adae87-71b3-4dfa-8eb0-3ad86c47e5a8\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr"
Nov 01 05:22:10.532581 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:10.532516    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54adae87-71b3-4dfa-8eb0-3ad86c47e5a8-istiod-ca-cert\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr\" (UID: \"54adae87-71b3-4dfa-8eb0-3ad86c47e5a8\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr"
Nov 01 05:22:10.532581 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:10.532546    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54adae87-71b3-4dfa-8eb0-3ad86c47e5a8-workload-socket\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr\" (UID: \"54adae87-71b3-4dfa-8eb0-3ad86c47e5a8\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr"
Nov 01 05:22:10.532676 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:10.532611    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54adae87-71b3-4dfa-8eb0-3ad86c47e5a8-istio-data\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr\" (UID: \"54adae87-71b3-4dfa-8eb0-3ad86c47e5a8\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr"
Nov 01 05:22:10.532732 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:10.532705    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54adae87-71b3-4dfa-8eb0-3ad86c47e5a8-credential-socket\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr\" (UID: \"54adae87-71b3-4dfa-8eb0-3ad86c47e5a8\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr"
Nov 01 05:22:10.532948 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:10.532753    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54adae87-71b3-4dfa-8eb0-3ad86c47e5a8-workload-certs\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr\" (UID: \"54adae87-71b3-4dfa-8eb0-3ad86c47e5a8\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr"
Nov 01 05:22:10.533112 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:10.533089    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54adae87-71b3-4dfa-8eb0-3ad86c47e5a8-istiod-ca-cert\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr\" (UID: \"54adae87-71b3-4dfa-8eb0-3ad86c47e5a8\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr"
Nov 01 05:22:10.534445 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:10.534424    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54adae87-71b3-4dfa-8eb0-3ad86c47e5a8-istio-envoy\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr\" (UID: \"54adae87-71b3-4dfa-8eb0-3ad86c47e5a8\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr"
Nov 01 05:22:10.534669 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:10.534654    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54adae87-71b3-4dfa-8eb0-3ad86c47e5a8-istio-podinfo\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr\" (UID: \"54adae87-71b3-4dfa-8eb0-3ad86c47e5a8\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr"
Nov 01 05:22:10.593078 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:10.593054    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54adae87-71b3-4dfa-8eb0-3ad86c47e5a8-istio-token\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr\" (UID: \"54adae87-71b3-4dfa-8eb0-3ad86c47e5a8\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr"
Nov 01 05:22:10.593402 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:10.593378    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzsvx\" (UniqueName: \"kubernetes.io/projected/54adae87-71b3-4dfa-8eb0-3ad86c47e5a8-kube-api-access-mzsvx\") pod \"e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr\" (UID: \"54adae87-71b3-4dfa-8eb0-3ad86c47e5a8\") " pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr"
Nov 01 05:22:10.664748 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:10.664721    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr"
Nov 01 05:22:11.093977 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:11.093952    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr"]
Nov 01 05:22:11.096800 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:22:11.096772    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54adae87_71b3_4dfa_8eb0_3ad86c47e5a8.slice/crio-4b1c0fb2639165866065c303ba3166e59f1452fec7ed5dac66d3b4af9464a968 WatchSource:0}: Error finding container 4b1c0fb2639165866065c303ba3166e59f1452fec7ed5dac66d3b4af9464a968: Status 404 returned error can't find the container with id 4b1c0fb2639165866065c303ba3166e59f1452fec7ed5dac66d3b4af9464a968
Nov 01 05:22:11.098978 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:11.098946    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:22:11.099098 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:11.099018    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:22:11.099098 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:11.099061    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:22:11.832342 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:11.832314    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr" event={"ID":"54adae87-71b3-4dfa-8eb0-3ad86c47e5a8","Type":"ContainerStarted","Data":"da06ff24d08604bb2d917b84c2b344bab48dc489f548c0386cf59c466f69b16e"}
Nov 01 05:22:11.832845 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:11.832361    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr" event={"ID":"54adae87-71b3-4dfa-8eb0-3ad86c47e5a8","Type":"ContainerStarted","Data":"4b1c0fb2639165866065c303ba3166e59f1452fec7ed5dac66d3b4af9464a968"}
Nov 01 05:22:11.948266 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:11.948213    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:22:11.952347 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:11.952302    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr" podStartSLOduration=1.952288905 podStartE2EDuration="1.952288905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:22:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:22:11.950600621 +0000 UTC m=+1110.021397365" watchObservedRunningTime="2026-11-01 05:22:11.952288905 +0000 UTC m=+1110.023085625"
Nov 01 05:22:12.128530 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:12.123154    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:22:12.664886 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:12.664857    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr"
Nov 01 05:22:12.666352 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:12.666323    5689 patch_prober.go:29] interesting pod/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Nov 01 05:22:12.666536 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:12.666390    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr" podUID="54adae87-71b3-4dfa-8eb0-3ad86c47e5a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Nov 01 05:22:13.665024 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:13.664988    5689 patch_prober.go:29] interesting pod/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Nov 01 05:22:13.665533 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:13.665056    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr" podUID="54adae87-71b3-4dfa-8eb0-3ad86c47e5a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Nov 01 05:22:14.620005 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:14.619973    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:22:14.687478 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:14.682424    5689 patch_prober.go:29] interesting pod/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Nov 01 05:22:14.687478 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:14.682502    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr" podUID="54adae87-71b3-4dfa-8eb0-3ad86c47e5a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Nov 01 05:22:15.667204 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:15.666699    5689 patch_prober.go:29] interesting pod/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Nov 01 05:22:15.667204 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:15.666790    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr" podUID="54adae87-71b3-4dfa-8eb0-3ad86c47e5a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Nov 01 05:22:16.665890 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:16.665852    5689 patch_prober.go:29] interesting pod/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Nov 01 05:22:16.666514 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:16.666482    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr" podUID="54adae87-71b3-4dfa-8eb0-3ad86c47e5a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Nov 01 05:22:17.353113 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:17.353079    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:22:17.665120 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:17.665087    5689 patch_prober.go:29] interesting pod/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Nov 01 05:22:17.665307 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:17.665237    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr" podUID="54adae87-71b3-4dfa-8eb0-3ad86c47e5a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Nov 01 05:22:18.976606 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:18.976571    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr"
Nov 01 05:22:18.976999 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:18.976974    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr"
Nov 01 05:22:18.977903 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:18.977878    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr"
Nov 01 05:22:20.889463 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:20.889392    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:22:21.396209 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:21.396177    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:22:22.452626 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:22.452590    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2864b6-669cd47975-bfwj8"]
Nov 01 05:22:22.458336 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:22.458314    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2864b6-669cd47975-bfwj8"
Nov 01 05:22:22.512761 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:22.512693    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-2864b6\""
Nov 01 05:22:22.515755 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:22.515731    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2864b6-669cd47975-bfwj8"]
Nov 01 05:22:22.560492 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:22.560463    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3642df08-0365-4562-b5f0-e480f50c2761-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2864b6-669cd47975-bfwj8\" (UID: \"3642df08-0365-4562-b5f0-e480f50c2761\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2864b6-669cd47975-bfwj8"
Nov 01 05:22:22.560704 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:22.560687    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjw7q\" (UniqueName: \"kubernetes.io/projected/3642df08-0365-4562-b5f0-e480f50c2761-kube-api-access-rjw7q\") pod \"payload-pre-processing-e2e-ipp-a-2864b6-669cd47975-bfwj8\" (UID: \"3642df08-0365-4562-b5f0-e480f50c2761\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2864b6-669cd47975-bfwj8"
Nov 01 05:22:22.562185 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:22.562168    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-2864b6-dockercfg-ht9wj\""
Nov 01 05:22:22.662073 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:22.662041    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3642df08-0365-4562-b5f0-e480f50c2761-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2864b6-669cd47975-bfwj8\" (UID: \"3642df08-0365-4562-b5f0-e480f50c2761\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2864b6-669cd47975-bfwj8"
Nov 01 05:22:22.662239 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:22.662223    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjw7q\" (UniqueName: \"kubernetes.io/projected/3642df08-0365-4562-b5f0-e480f50c2761-kube-api-access-rjw7q\") pod \"payload-pre-processing-e2e-ipp-a-2864b6-669cd47975-bfwj8\" (UID: \"3642df08-0365-4562-b5f0-e480f50c2761\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2864b6-669cd47975-bfwj8"
Nov 01 05:22:22.662716 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:22.662700    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3642df08-0365-4562-b5f0-e480f50c2761-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2864b6-669cd47975-bfwj8\" (UID: \"3642df08-0365-4562-b5f0-e480f50c2761\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2864b6-669cd47975-bfwj8"
Nov 01 05:22:22.723157 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:22.723102    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjw7q\" (UniqueName: \"kubernetes.io/projected/3642df08-0365-4562-b5f0-e480f50c2761-kube-api-access-rjw7q\") pod \"payload-pre-processing-e2e-ipp-a-2864b6-669cd47975-bfwj8\" (UID: \"3642df08-0365-4562-b5f0-e480f50c2761\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2864b6-669cd47975-bfwj8"
Nov 01 05:22:22.773095 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:22.773071    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2864b6-669cd47975-bfwj8"
Nov 01 05:22:23.148492 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:23.148446    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2864b6-669cd47975-bfwj8"]
Nov 01 05:22:23.151201 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:22:23.151178    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3642df08_0365_4562_b5f0_e480f50c2761.slice/crio-6f56eb3e0a459047e25eb7eed1d3d729ff8f20e9ef5c60598405a3cadf2b47d5 WatchSource:0}: Error finding container 6f56eb3e0a459047e25eb7eed1d3d729ff8f20e9ef5c60598405a3cadf2b47d5: Status 404 returned error can't find the container with id 6f56eb3e0a459047e25eb7eed1d3d729ff8f20e9ef5c60598405a3cadf2b47d5
Nov 01 05:22:23.258269 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:23.258246    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2864b6-6d848878b6-zcftx"]
Nov 01 05:22:23.263104 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:23.263088    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2864b6-6d848878b6-zcftx"
Nov 01 05:22:23.322602 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:23.322574    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2864b6-6d848878b6-zcftx"]
Nov 01 05:22:23.369643 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:23.369619    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/390739de-f035-41a9-b2d9-ff8664766ba6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2864b6-6d848878b6-zcftx\" (UID: \"390739de-f035-41a9-b2d9-ff8664766ba6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2864b6-6d848878b6-zcftx"
Nov 01 05:22:23.369827 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:23.369813    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh48d\" (UniqueName: \"kubernetes.io/projected/390739de-f035-41a9-b2d9-ff8664766ba6-kube-api-access-rh48d\") pod \"payload-processing-e2e-ipp-a-2864b6-6d848878b6-zcftx\" (UID: \"390739de-f035-41a9-b2d9-ff8664766ba6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2864b6-6d848878b6-zcftx"
Nov 01 05:22:23.470441 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:23.470341    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/390739de-f035-41a9-b2d9-ff8664766ba6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2864b6-6d848878b6-zcftx\" (UID: \"390739de-f035-41a9-b2d9-ff8664766ba6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2864b6-6d848878b6-zcftx"
Nov 01 05:22:23.470789 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:23.470547    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh48d\" (UniqueName: \"kubernetes.io/projected/390739de-f035-41a9-b2d9-ff8664766ba6-kube-api-access-rh48d\") pod \"payload-processing-e2e-ipp-a-2864b6-6d848878b6-zcftx\" (UID: \"390739de-f035-41a9-b2d9-ff8664766ba6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2864b6-6d848878b6-zcftx"
Nov 01 05:22:23.471045 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:23.471027    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/390739de-f035-41a9-b2d9-ff8664766ba6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2864b6-6d848878b6-zcftx\" (UID: \"390739de-f035-41a9-b2d9-ff8664766ba6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2864b6-6d848878b6-zcftx"
Nov 01 05:22:23.530968 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:23.530943    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh48d\" (UniqueName: \"kubernetes.io/projected/390739de-f035-41a9-b2d9-ff8664766ba6-kube-api-access-rh48d\") pod \"payload-processing-e2e-ipp-a-2864b6-6d848878b6-zcftx\" (UID: \"390739de-f035-41a9-b2d9-ff8664766ba6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2864b6-6d848878b6-zcftx"
Nov 01 05:22:23.576717 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:23.576692    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2864b6-6d848878b6-zcftx"
Nov 01 05:22:23.951229 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:23.951099    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2864b6-6d848878b6-zcftx"]
Nov 01 05:22:23.954001 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:22:23.953970    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod390739de_f035_41a9_b2d9_ff8664766ba6.slice/crio-2e21a8a185541d60d8d0f2c3d5c686e2d4d1db1750de631bf68049029e84b558 WatchSource:0}: Error finding container 2e21a8a185541d60d8d0f2c3d5c686e2d4d1db1750de631bf68049029e84b558: Status 404 returned error can't find the container with id 2e21a8a185541d60d8d0f2c3d5c686e2d4d1db1750de631bf68049029e84b558
Nov 01 05:22:23.987645 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:23.987618    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2864b6-6d848878b6-zcftx" event={"ID":"390739de-f035-41a9-b2d9-ff8664766ba6","Type":"ContainerStarted","Data":"2e21a8a185541d60d8d0f2c3d5c686e2d4d1db1750de631bf68049029e84b558"}
Nov 01 05:22:23.988811 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:23.988789    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2864b6-669cd47975-bfwj8" event={"ID":"3642df08-0365-4562-b5f0-e480f50c2761","Type":"ContainerStarted","Data":"6f56eb3e0a459047e25eb7eed1d3d729ff8f20e9ef5c60598405a3cadf2b47d5"}
Nov 01 05:22:23.994044 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:23.994023    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:22:26.014158 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:26.014066    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2864b6-6d848878b6-zcftx" event={"ID":"390739de-f035-41a9-b2d9-ff8664766ba6","Type":"ContainerStarted","Data":"f780cbd612d523e2f357ecb82488390a61f2a852c80746ad8f68c6244a61a50e"}
Nov 01 05:22:26.014158 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:26.014142    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2864b6-6d848878b6-zcftx"
Nov 01 05:22:26.015625 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:26.015601    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2864b6-669cd47975-bfwj8" event={"ID":"3642df08-0365-4562-b5f0-e480f50c2761","Type":"ContainerStarted","Data":"6d3fddc8e4e1fdd317a6b08d93b90f7fb0182298f8723ef9446efae524c6b4d5"}
Nov 01 05:22:26.015737 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:26.015718    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2864b6-669cd47975-bfwj8"
Nov 01 05:22:26.131319 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:26.131263    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-2864b6-6d848878b6-zcftx" podStartSLOduration=1.318112277 podStartE2EDuration="3.131250476s" totalImagesPullingTime="1.813138199s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:22:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:22:26.129057192 +0000 UTC m=+1124.199853927" watchObservedRunningTime="2026-11-01 05:22:26.131250476 +0000 UTC m=+1124.202047225"
Nov 01 05:22:26.258740 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:26.258545    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2864b6-669cd47975-bfwj8" podStartSLOduration=1.7032537300000001 podStartE2EDuration="4.258525598s" totalImagesPullingTime="2.555271868s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:22:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:22:26.256650876 +0000 UTC m=+1124.327447598" watchObservedRunningTime="2026-11-01 05:22:26.258525598 +0000 UTC m=+1124.329322321"
Nov 01 05:22:28.591038 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:28.591002    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:22:28.765556 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:28.765525    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:22:28.956310 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:28.956274    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:22:29.179604 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:29.179568    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:22:30.069771 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:22:30.069737    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:22:31.393482 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:31.393428    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:22:31.881679 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:31.881644    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:22:33.646008 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:33.645972    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:22:33.826498 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:33.826424    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:22:34.008805 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:34.008762    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:22:37.023869 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:37.023832    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2864b6-669cd47975-bfwj8"
Nov 01 05:22:37.024247 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:37.023910    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2864b6-6d848878b6-zcftx"
Nov 01 05:22:38.196436 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:38.196400    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:22:38.837586 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:38.837555    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:22:44.045247 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:44.045212    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:22:46.207574 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:46.207534    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2ea125-5cc577d6b5-p45rd"]
Nov 01 05:22:46.625783 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:46.625702    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:22:46.803146 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:46.803111    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:22:48.258908 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:48.258871    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:22:48.934482 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:48.929404    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:22:50.044072 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:50.044035    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:22:51.213479 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:51.211516    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-2ea125-5cc577d6b5-p45rd" podUID="731cd5ef-5a2a-4771-8067-75b133315fbf" containerName="payload-processing" containerID="cri-o://38148b014b4d51fd0330081708bf86333cd0f87d4b882076f7fd320c3eec3c57" gracePeriod=25
Nov 01 05:22:51.427478 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:51.413561    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-2ea125-5cc577d6b5-p45rd" podUID="731cd5ef-5a2a-4771-8067-75b133315fbf" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 05:22:52.900410 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:52.900375    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:22:54.640807 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:54.640777    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-m5kl4"]
Nov 01 05:22:54.647586 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:54.647562    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-m5kl4"
Nov 01 05:22:54.752496 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:54.752439    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-m5kl4"]
Nov 01 05:22:54.828526 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:54.828497    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ee14c22-76b6-4c41-bb64-c374e08328f1-workload-socket\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-m5kl4\" (UID: \"3ee14c22-76b6-4c41-bb64-c374e08328f1\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-m5kl4"
Nov 01 05:22:54.828688 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:54.828532    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ee14c22-76b6-4c41-bb64-c374e08328f1-istio-data\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-m5kl4\" (UID: \"3ee14c22-76b6-4c41-bb64-c374e08328f1\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-m5kl4"
Nov 01 05:22:54.828688 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:54.828563    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ee14c22-76b6-4c41-bb64-c374e08328f1-istiod-ca-cert\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-m5kl4\" (UID: \"3ee14c22-76b6-4c41-bb64-c374e08328f1\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-m5kl4"
Nov 01 05:22:54.828688 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:54.828642    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ee14c22-76b6-4c41-bb64-c374e08328f1-istio-token\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-m5kl4\" (UID: \"3ee14c22-76b6-4c41-bb64-c374e08328f1\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-m5kl4"
Nov 01 05:22:54.828688 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:54.828684    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4j9l\" (UniqueName: \"kubernetes.io/projected/3ee14c22-76b6-4c41-bb64-c374e08328f1-kube-api-access-t4j9l\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-m5kl4\" (UID: \"3ee14c22-76b6-4c41-bb64-c374e08328f1\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-m5kl4"
Nov 01 05:22:54.828905 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:54.828842    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ee14c22-76b6-4c41-bb64-c374e08328f1-credential-socket\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-m5kl4\" (UID: \"3ee14c22-76b6-4c41-bb64-c374e08328f1\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-m5kl4"
Nov 01 05:22:54.828905 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:54.828889    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ee14c22-76b6-4c41-bb64-c374e08328f1-workload-certs\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-m5kl4\" (UID: \"3ee14c22-76b6-4c41-bb64-c374e08328f1\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-m5kl4"
Nov 01 05:22:54.829046 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:54.829031    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ee14c22-76b6-4c41-bb64-c374e08328f1-istio-envoy\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-m5kl4\" (UID: \"3ee14c22-76b6-4c41-bb64-c374e08328f1\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-m5kl4"
Nov 01 05:22:54.829112 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:54.829079    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ee14c22-76b6-4c41-bb64-c374e08328f1-istio-podinfo\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-m5kl4\" (UID: \"3ee14c22-76b6-4c41-bb64-c374e08328f1\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-m5kl4"
Nov 01 05:22:54.901779 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:54.901747    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:22:54.932373 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:54.932332    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ee14c22-76b6-4c41-bb64-c374e08328f1-istio-envoy\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-m5kl4\" (UID: \"3ee14c22-76b6-4c41-bb64-c374e08328f1\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-m5kl4"
Nov 01 05:22:54.933635 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:54.933607    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ee14c22-76b6-4c41-bb64-c374e08328f1-istio-podinfo\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-m5kl4\" (UID: \"3ee14c22-76b6-4c41-bb64-c374e08328f1\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-m5kl4"
Nov 01 05:22:54.934336 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:54.934314    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ee14c22-76b6-4c41-bb64-c374e08328f1-workload-socket\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-m5kl4\" (UID: \"3ee14c22-76b6-4c41-bb64-c374e08328f1\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-m5kl4"
Nov 01 05:22:54.934502 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:54.934486    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ee14c22-76b6-4c41-bb64-c374e08328f1-istio-data\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-m5kl4\" (UID: \"3ee14c22-76b6-4c41-bb64-c374e08328f1\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-m5kl4"
Nov 01 05:22:54.934622 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:54.934608    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ee14c22-76b6-4c41-bb64-c374e08328f1-istiod-ca-cert\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-m5kl4\" (UID: \"3ee14c22-76b6-4c41-bb64-c374e08328f1\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-m5kl4"
Nov 01 05:22:54.934757 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:54.934742    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ee14c22-76b6-4c41-bb64-c374e08328f1-istio-token\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-m5kl4\" (UID: \"3ee14c22-76b6-4c41-bb64-c374e08328f1\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-m5kl4"
Nov 01 05:22:54.934910 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:54.934892    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4j9l\" (UniqueName: \"kubernetes.io/projected/3ee14c22-76b6-4c41-bb64-c374e08328f1-kube-api-access-t4j9l\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-m5kl4\" (UID: \"3ee14c22-76b6-4c41-bb64-c374e08328f1\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-m5kl4"
Nov 01 05:22:54.935619 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:54.935593    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ee14c22-76b6-4c41-bb64-c374e08328f1-istiod-ca-cert\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-m5kl4\" (UID: \"3ee14c22-76b6-4c41-bb64-c374e08328f1\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-m5kl4"
Nov 01 05:22:54.935719 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:54.934803    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ee14c22-76b6-4c41-bb64-c374e08328f1-workload-socket\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-m5kl4\" (UID: \"3ee14c22-76b6-4c41-bb64-c374e08328f1\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-m5kl4"
Nov 01 05:22:54.935719 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:54.935598    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ee14c22-76b6-4c41-bb64-c374e08328f1-credential-socket\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-m5kl4\" (UID: \"3ee14c22-76b6-4c41-bb64-c374e08328f1\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-m5kl4"
Nov 01 05:22:54.935719 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:54.935682    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ee14c22-76b6-4c41-bb64-c374e08328f1-workload-certs\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-m5kl4\" (UID: \"3ee14c22-76b6-4c41-bb64-c374e08328f1\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-m5kl4"
Nov 01 05:22:54.935889 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:54.935020    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ee14c22-76b6-4c41-bb64-c374e08328f1-istio-data\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-m5kl4\" (UID: \"3ee14c22-76b6-4c41-bb64-c374e08328f1\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-m5kl4"
Nov 01 05:22:54.937214 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:54.936070    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ee14c22-76b6-4c41-bb64-c374e08328f1-workload-certs\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-m5kl4\" (UID: \"3ee14c22-76b6-4c41-bb64-c374e08328f1\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-m5kl4"
Nov 01 05:22:54.944827 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:54.937732    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ee14c22-76b6-4c41-bb64-c374e08328f1-credential-socket\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-m5kl4\" (UID: \"3ee14c22-76b6-4c41-bb64-c374e08328f1\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-m5kl4"
Nov 01 05:22:54.952361 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:54.952329    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ee14c22-76b6-4c41-bb64-c374e08328f1-istio-envoy\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-m5kl4\" (UID: \"3ee14c22-76b6-4c41-bb64-c374e08328f1\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-m5kl4"
Nov 01 05:22:54.954095 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:54.954060    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ee14c22-76b6-4c41-bb64-c374e08328f1-istio-podinfo\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-m5kl4\" (UID: \"3ee14c22-76b6-4c41-bb64-c374e08328f1\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-m5kl4"
Nov 01 05:22:54.995938 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:54.995904    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ee14c22-76b6-4c41-bb64-c374e08328f1-istio-token\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-m5kl4\" (UID: \"3ee14c22-76b6-4c41-bb64-c374e08328f1\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-m5kl4"
Nov 01 05:22:54.996133 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:54.996116    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4j9l\" (UniqueName: \"kubernetes.io/projected/3ee14c22-76b6-4c41-bb64-c374e08328f1-kube-api-access-t4j9l\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-m5kl4\" (UID: \"3ee14c22-76b6-4c41-bb64-c374e08328f1\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-m5kl4"
Nov 01 05:22:55.261498 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:55.261410    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-m5kl4"
Nov 01 05:22:55.704331 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:55.704301    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-m5kl4"]
Nov 01 05:22:55.706064 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:22:55.706033    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ee14c22_76b6_4c41_bb64_c374e08328f1.slice/crio-4104beb1d7f6dfb7d8d3d5d13ae357ef30a21df188077e86e0623104f25cab34 WatchSource:0}: Error finding container 4104beb1d7f6dfb7d8d3d5d13ae357ef30a21df188077e86e0623104f25cab34: Status 404 returned error can't find the container with id 4104beb1d7f6dfb7d8d3d5d13ae357ef30a21df188077e86e0623104f25cab34
Nov 01 05:22:55.708652 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:55.708620    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:22:55.708726 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:55.708684    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:22:55.708726 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:55.708710    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:22:56.305741 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:56.305692    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-m5kl4" event={"ID":"3ee14c22-76b6-4c41-bb64-c374e08328f1","Type":"ContainerStarted","Data":"585ad222babf72524a00368bfbcdd7aa655936a25f4c1006fa4f69f713fcdb3f"}
Nov 01 05:22:56.305947 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:56.305774    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-m5kl4" event={"ID":"3ee14c22-76b6-4c41-bb64-c374e08328f1","Type":"ContainerStarted","Data":"4104beb1d7f6dfb7d8d3d5d13ae357ef30a21df188077e86e0623104f25cab34"}
Nov 01 05:22:56.502728 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:56.502659    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-m5kl4" podStartSLOduration=2.502640097 podStartE2EDuration="2.502640097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:22:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:22:56.474071457 +0000 UTC m=+1154.544868179" watchObservedRunningTime="2026-11-01 05:22:56.502640097 +0000 UTC m=+1154.573436818"
Nov 01 05:22:57.262212 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:57.262177    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-m5kl4"
Nov 01 05:22:57.263463 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:57.263433    5689 patch_prober.go:29] interesting pod/e2e-worker-w4-a48099-openshift-default-577864759-m5kl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Nov 01 05:22:57.263527 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:57.263493    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-m5kl4" podUID="3ee14c22-76b6-4c41-bb64-c374e08328f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Nov 01 05:22:58.262153 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:58.262117    5689 patch_prober.go:29] interesting pod/e2e-worker-w4-a48099-openshift-default-577864759-m5kl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Nov 01 05:22:58.262325 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:58.262180    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-m5kl4" podUID="3ee14c22-76b6-4c41-bb64-c374e08328f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Nov 01 05:22:59.052680 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:59.051491    5689 patch_prober.go:29] interesting pod/e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": read tcp 10.134.0.2:33434->10.134.0.66:15021: read: connection reset by peer" start-of-body=
Nov 01 05:22:59.052680 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:59.051554    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd" podUID="a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": read tcp 10.134.0.2:33434->10.134.0.66:15021: read: connection reset by peer"
Nov 01 05:22:59.262593 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:59.262511    5689 patch_prober.go:29] interesting pod/e2e-worker-w4-a48099-openshift-default-577864759-m5kl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Nov 01 05:22:59.262593 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:59.262568    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-m5kl4" podUID="3ee14c22-76b6-4c41-bb64-c374e08328f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Nov 01 05:22:59.325773 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:59.325700    5689 generic.go:356] "Generic (PLEG): container finished" podID="a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50" containerID="e471ff50c9e2245d141f675eba5be657e2eab2385dfd2d032238ae783a61319e" exitCode=0
Nov 01 05:22:59.325927 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:59.325771    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd" event={"ID":"a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50","Type":"ContainerDied","Data":"e471ff50c9e2245d141f675eba5be657e2eab2385dfd2d032238ae783a61319e"}
Nov 01 05:22:59.326161 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:59.326134    5689 scope.go:122] "RemoveContainer" containerID="e471ff50c9e2245d141f675eba5be657e2eab2385dfd2d032238ae783a61319e"
Nov 01 05:22:59.327512 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:59.327487    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:22:59.327570 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:59.327556    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:22:59.327605 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:59.327592    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:22:59.725479 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:59.723620    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:22:59.970217 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:22:59.970183    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:23:00.069497 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:23:00.069396    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:23:00.262125 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:00.262082    5689 patch_prober.go:29] interesting pod/e2e-worker-w4-a48099-openshift-default-577864759-m5kl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Nov 01 05:23:00.262320 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:00.262129    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-m5kl4" podUID="3ee14c22-76b6-4c41-bb64-c374e08328f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Nov 01 05:23:00.338532 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:00.337728    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd" event={"ID":"a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50","Type":"ContainerStarted","Data":"765d1c81e31cc0bfb6a85bba43660d16aad48259c7a496acfe3fcdf60fe38e91"}
Nov 01 05:23:00.437903 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:00.437804    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd"
Nov 01 05:23:00.438060 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:00.437950    5689 patch_prober.go:29] interesting pod/e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 01 05:23:00.438060 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:00.438006    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd" podUID="a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 01 05:23:01.261951 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:01.261916    5689 patch_prober.go:29] interesting pod/e2e-worker-w4-a48099-openshift-default-577864759-m5kl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Nov 01 05:23:01.262126 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:01.261963    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-m5kl4" podUID="3ee14c22-76b6-4c41-bb64-c374e08328f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Nov 01 05:23:01.424379 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:01.422557    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-2ea125-5cc577d6b5-p45rd" podUID="731cd5ef-5a2a-4771-8067-75b133315fbf" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 05:23:01.438400 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:01.438361    5689 patch_prober.go:29] interesting pod/e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 01 05:23:01.438591 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:01.438419    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd" podUID="a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 01 05:23:02.266077 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:02.266042    5689 patch_prober.go:29] interesting pod/e2e-worker-w4-a48099-openshift-default-577864759-m5kl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Nov 01 05:23:02.266261 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:02.266110    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-m5kl4" podUID="3ee14c22-76b6-4c41-bb64-c374e08328f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Nov 01 05:23:02.438906 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:02.438871    5689 patch_prober.go:29] interesting pod/e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 01 05:23:02.439366 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:02.438925    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd" podUID="a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 01 05:23:03.266121 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:03.266094    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-m5kl4"
Nov 01 05:23:03.266403 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:03.266383    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-m5kl4"
Nov 01 05:23:03.267201 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:03.267183    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-m5kl4"
Nov 01 05:23:03.437985 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:03.437902    5689 patch_prober.go:29] interesting pod/e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 01 05:23:03.437985 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:03.437954    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd" podUID="a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 01 05:23:04.437547 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:04.437516    5689 patch_prober.go:29] interesting pod/e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 01 05:23:04.437942 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:04.437579    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd" podUID="a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 01 05:23:04.754327 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:04.754191    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:23:05.438792 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:05.438751    5689 patch_prober.go:29] interesting pod/e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 01 05:23:05.439262 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:05.438804    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd" podUID="a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 01 05:23:06.441419 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:06.441385    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd"
Nov 01 05:23:06.441916 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:06.441807    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd"
Nov 01 05:23:06.442357 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:06.442336    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd"
Nov 01 05:23:09.189857 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:09.189820    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:23:11.433487 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:23:11.429486    5689 logging.go:55] [core] [Channel #428 SubChannel #429]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.47:50052", ServerName: "10.134.0.47:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.47:50052: connect: connection refused"
Nov 01 05:23:11.488249 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:11.488157    5689 generic.go:356] "Generic (PLEG): container finished" podID="731cd5ef-5a2a-4771-8067-75b133315fbf" containerID="38148b014b4d51fd0330081708bf86333cd0f87d4b882076f7fd320c3eec3c57" exitCode=0
Nov 01 05:23:11.492707 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:11.492671    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2ea125-5cc577d6b5-p45rd" event={"ID":"731cd5ef-5a2a-4771-8067-75b133315fbf","Type":"ContainerDied","Data":"38148b014b4d51fd0330081708bf86333cd0f87d4b882076f7fd320c3eec3c57"}
Nov 01 05:23:11.603654 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:11.603617    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:23:12.100804 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:12.100760    5689 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-3d70ff/simulator-w2-3d70ff-kserve-898c5fb4f-t5bzk" podUID="05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 05:23:12.121672 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:12.121359    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2ea125-5cc577d6b5-p45rd"
Nov 01 05:23:12.301306 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:12.301267    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/731cd5ef-5a2a-4771-8067-75b133315fbf-kube-api-access-j4wbm\" (UniqueName: \"kubernetes.io/projected/731cd5ef-5a2a-4771-8067-75b133315fbf-kube-api-access-j4wbm\") pod \"731cd5ef-5a2a-4771-8067-75b133315fbf\" (UID: \"731cd5ef-5a2a-4771-8067-75b133315fbf\") "
Nov 01 05:23:12.301521 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:12.301489    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/731cd5ef-5a2a-4771-8067-75b133315fbf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/731cd5ef-5a2a-4771-8067-75b133315fbf-plugins-config-volume\") pod \"731cd5ef-5a2a-4771-8067-75b133315fbf\" (UID: \"731cd5ef-5a2a-4771-8067-75b133315fbf\") "
Nov 01 05:23:12.301834 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:12.301809    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/731cd5ef-5a2a-4771-8067-75b133315fbf-plugins-config-volume" pod "731cd5ef-5a2a-4771-8067-75b133315fbf" (UID: "731cd5ef-5a2a-4771-8067-75b133315fbf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:23:12.301988 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:12.301970    5689 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/731cd5ef-5a2a-4771-8067-75b133315fbf-plugins-config-volume\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:23:12.303556 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:12.303532    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/731cd5ef-5a2a-4771-8067-75b133315fbf-kube-api-access-j4wbm" pod "731cd5ef-5a2a-4771-8067-75b133315fbf" (UID: "731cd5ef-5a2a-4771-8067-75b133315fbf"). InnerVolumeSpecName "kube-api-access-j4wbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:23:12.402719 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:12.402692    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4wbm\" (UniqueName: \"kubernetes.io/projected/731cd5ef-5a2a-4771-8067-75b133315fbf-kube-api-access-j4wbm\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:23:12.406738 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:12.406705    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-2ea125-5cc577d6b5-p45rd" podUID="731cd5ef-5a2a-4771-8067-75b133315fbf" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.47:50052\" within 1s: context deadline exceeded"
Nov 01 05:23:12.406840 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:12.406791    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-2ea125-5cc577d6b5-p45rd"
Nov 01 05:23:12.500313 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:12.500287    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2ea125-5cc577d6b5-p45rd"
Nov 01 05:23:12.500810 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:12.500311    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2ea125-5cc577d6b5-p45rd" event={"ID":"731cd5ef-5a2a-4771-8067-75b133315fbf","Type":"ContainerDied","Data":"67bc233fef281420d2ba8975aa358b31f536c8281de8b124823e96f32d19c533"}
Nov 01 05:23:12.500810 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:12.500400    5689 scope.go:122] "RemoveContainer" containerID="38148b014b4d51fd0330081708bf86333cd0f87d4b882076f7fd320c3eec3c57"
Nov 01 05:23:12.665069 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:12.665036    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2ea125-5cc577d6b5-p45rd"]
Nov 01 05:23:12.670493 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:12.670445    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2ea125-5cc577d6b5-p45rd"]
Nov 01 05:23:14.069059 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:14.069028    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="731cd5ef-5a2a-4771-8067-75b133315fbf" path="/var/lib/kubelet/pods/731cd5ef-5a2a-4771-8067-75b133315fbf/volumes"
Nov 01 05:23:16.417384 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:16.417346    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:23:21.203223 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:21.203150    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:23:26.012151 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:26.012117    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:23:28.172482 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:28.171953    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:23:28.468860 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:28.468763    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3d70ff/simulator-w2-3d70ff-kserve-898c5fb4f-t5bzk" podUID="05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 05:23:28.488476 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:28.488409    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a48099/simulator-w4-a48099-kserve-7df86f7d-rdlrm" podUID="12c3ad65-a804-429f-ab42-9cc7b2433855" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 05:23:30.068754 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:23:30.068720    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:23:30.698422 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:30.698391    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5451cb-794c5878c5-4mnnq"]
Nov 01 05:23:30.992705 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:30.992626    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:23:31.177221 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:31.176335    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:23:31.439941 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:31.439887    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:23:35.699539 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:35.699482    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5451cb-794c5878c5-4mnnq" podUID="61f28ebd-0530-4058-9f4b-a3b17b6c04aa" containerName="payload-pre-processing" containerID="cri-o://60630d5c73fd01c144f293e047ea79a077be65de9c9b40a0fb95d5dc3c18ada9" gracePeriod=25
Nov 01 05:23:35.966556 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:35.966502    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5451cb-794c5878c5-4mnnq"
Nov 01 05:23:36.067917 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:36.067892    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61f28ebd-0530-4058-9f4b-a3b17b6c04aa-kube-api-access-k57mn\" (UniqueName: \"kubernetes.io/projected/61f28ebd-0530-4058-9f4b-a3b17b6c04aa-kube-api-access-k57mn\") pod \"61f28ebd-0530-4058-9f4b-a3b17b6c04aa\" (UID: \"61f28ebd-0530-4058-9f4b-a3b17b6c04aa\") "
Nov 01 05:23:36.068078 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:36.067931    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61f28ebd-0530-4058-9f4b-a3b17b6c04aa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61f28ebd-0530-4058-9f4b-a3b17b6c04aa-plugins-config-volume\") pod \"61f28ebd-0530-4058-9f4b-a3b17b6c04aa\" (UID: \"61f28ebd-0530-4058-9f4b-a3b17b6c04aa\") "
Nov 01 05:23:36.068317 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:36.068297    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61f28ebd-0530-4058-9f4b-a3b17b6c04aa-plugins-config-volume" pod "61f28ebd-0530-4058-9f4b-a3b17b6c04aa" (UID: "61f28ebd-0530-4058-9f4b-a3b17b6c04aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:23:36.070238 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:36.070213    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61f28ebd-0530-4058-9f4b-a3b17b6c04aa-kube-api-access-k57mn" pod "61f28ebd-0530-4058-9f4b-a3b17b6c04aa" (UID: "61f28ebd-0530-4058-9f4b-a3b17b6c04aa"). InnerVolumeSpecName "kube-api-access-k57mn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:23:36.169220 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:36.169186    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k57mn\" (UniqueName: \"kubernetes.io/projected/61f28ebd-0530-4058-9f4b-a3b17b6c04aa-kube-api-access-k57mn\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:23:36.169220 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:36.169223    5689 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61f28ebd-0530-4058-9f4b-a3b17b6c04aa-plugins-config-volume\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:23:36.792319 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:36.792286    5689 generic.go:356] "Generic (PLEG): container finished" podID="61f28ebd-0530-4058-9f4b-a3b17b6c04aa" containerID="60630d5c73fd01c144f293e047ea79a077be65de9c9b40a0fb95d5dc3c18ada9" exitCode=0
Nov 01 05:23:36.792748 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:36.792392    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5451cb-794c5878c5-4mnnq"
Nov 01 05:23:36.792748 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:36.792422    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5451cb-794c5878c5-4mnnq" event={"ID":"61f28ebd-0530-4058-9f4b-a3b17b6c04aa","Type":"ContainerDied","Data":"60630d5c73fd01c144f293e047ea79a077be65de9c9b40a0fb95d5dc3c18ada9"}
Nov 01 05:23:36.792748 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:36.792484    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5451cb-794c5878c5-4mnnq" event={"ID":"61f28ebd-0530-4058-9f4b-a3b17b6c04aa","Type":"ContainerDied","Data":"c548a59e6b4d63415ebe4d6f900ed88f5dc78b6a2367e78b4017c0d81f8fe8c5"}
Nov 01 05:23:36.792748 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:36.792507    5689 scope.go:122] "RemoveContainer" containerID="60630d5c73fd01c144f293e047ea79a077be65de9c9b40a0fb95d5dc3c18ada9"
Nov 01 05:23:36.803356 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:36.803339    5689 scope.go:122] "RemoveContainer" containerID="60630d5c73fd01c144f293e047ea79a077be65de9c9b40a0fb95d5dc3c18ada9"
Nov 01 05:23:36.803636 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:23:36.803614    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60630d5c73fd01c144f293e047ea79a077be65de9c9b40a0fb95d5dc3c18ada9\": container with ID starting with 60630d5c73fd01c144f293e047ea79a077be65de9c9b40a0fb95d5dc3c18ada9 not found: ID does not exist" containerID="60630d5c73fd01c144f293e047ea79a077be65de9c9b40a0fb95d5dc3c18ada9"
Nov 01 05:23:36.803710 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:36.803647    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60630d5c73fd01c144f293e047ea79a077be65de9c9b40a0fb95d5dc3c18ada9"} err="failed to get container status \"60630d5c73fd01c144f293e047ea79a077be65de9c9b40a0fb95d5dc3c18ada9\": rpc error: code = NotFound desc = could not find container \"60630d5c73fd01c144f293e047ea79a077be65de9c9b40a0fb95d5dc3c18ada9\": container with ID starting with 60630d5c73fd01c144f293e047ea79a077be65de9c9b40a0fb95d5dc3c18ada9 not found: ID does not exist"
Nov 01 05:23:36.817088 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:23:36.817066    5689 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61f28ebd_0530_4058_9f4b_a3b17b6c04aa.slice\": RecentStats: unable to find data in memory cache]"
Nov 01 05:23:36.955714 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:36.955681    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5451cb-794c5878c5-4mnnq"]
Nov 01 05:23:37.005026 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:37.004996    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5451cb-794c5878c5-4mnnq"]
Nov 01 05:23:37.219635 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:37.219599    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:23:38.067222 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:38.067191    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61f28ebd-0530-4058-9f4b-a3b17b6c04aa" path="/var/lib/kubelet/pods/61f28ebd-0530-4058-9f4b-a3b17b6c04aa/volumes"
Nov 01 05:23:39.640164 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:39.640128    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:23:39.861186 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:39.861154    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:23:42.300001 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:42.299964    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:23:43.060796 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:43.060761    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:23:43.241300 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:43.241269    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:23:44.608273 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:44.608239    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:23:45.214891 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:45.214859    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:23:46.045672 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:46.045633    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:23:47.350671 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:47.350638    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:23:52.303465 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:52.303424    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:23:52.530589 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:52.530550    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:23:53.349784 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:53.347098    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-af8051-5599f954b9-g8vp8"]
Nov 01 05:23:53.756330 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:53.756297    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:23:53.940912 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:53.940875    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:23:58.215572 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:58.215534    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:23:58.347570 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:58.347518    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-af8051-5599f954b9-g8vp8" podUID="bc3118cb-d425-4294-a46c-12a554343918" containerName="payload-processing" containerID="cri-o://8b6e6819a781276f924c22ea6a6f6a39e4fbee54ea16d58a4dbb585c5d4a16f2" gracePeriod=25
Nov 01 05:23:58.395550 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:58.395518    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-af8051-8f8c87946-qzrqg"]
Nov 01 05:23:58.396007 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:58.395973    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af8051-8f8c87946-qzrqg" podUID="b535e368-228c-49c3-b014-445acbad9d0d" containerName="maas-api" containerID="cri-o://5b4eac7be11fda86a4ac565c99e868b23848fbd275883a42eeed09b577f059cf" gracePeriod=30
Nov 01 05:23:58.534574 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:58.497262    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:23:59.062016 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:59.061743    5689 generic.go:356] "Generic (PLEG): container finished" podID="b535e368-228c-49c3-b014-445acbad9d0d" containerID="5b4eac7be11fda86a4ac565c99e868b23848fbd275883a42eeed09b577f059cf" exitCode=0
Nov 01 05:23:59.062016 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:59.061902    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af8051-8f8c87946-qzrqg" event={"ID":"b535e368-228c-49c3-b014-445acbad9d0d","Type":"ContainerDied","Data":"5b4eac7be11fda86a4ac565c99e868b23848fbd275883a42eeed09b577f059cf"}
Nov 01 05:23:59.505057 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:59.504536    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a48099/simulator-w4-a48099-kserve-7df86f7d-rdlrm" podUID="12c3ad65-a804-429f-ab42-9cc7b2433855" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 05:23:59.582953 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:59.582665    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af8051-8f8c87946-qzrqg"
Nov 01 05:23:59.731523 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:59.731486    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b535e368-228c-49c3-b014-445acbad9d0d-kube-api-access-tlx8k\" (UniqueName: \"kubernetes.io/projected/b535e368-228c-49c3-b014-445acbad9d0d-kube-api-access-tlx8k\") pod \"b535e368-228c-49c3-b014-445acbad9d0d\" (UID: \"b535e368-228c-49c3-b014-445acbad9d0d\") "
Nov 01 05:23:59.731831 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:59.731807    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b535e368-228c-49c3-b014-445acbad9d0d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b535e368-228c-49c3-b014-445acbad9d0d-metrics-certs\") pod \"b535e368-228c-49c3-b014-445acbad9d0d\" (UID: \"b535e368-228c-49c3-b014-445acbad9d0d\") "
Nov 01 05:23:59.732050 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:59.732031    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b535e368-228c-49c3-b014-445acbad9d0d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b535e368-228c-49c3-b014-445acbad9d0d-maas-api-tls\") pod \"b535e368-228c-49c3-b014-445acbad9d0d\" (UID: \"b535e368-228c-49c3-b014-445acbad9d0d\") "
Nov 01 05:23:59.732197 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:59.732179    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b535e368-228c-49c3-b014-445acbad9d0d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b535e368-228c-49c3-b014-445acbad9d0d-openshift-service-ca\") pod \"b535e368-228c-49c3-b014-445acbad9d0d\" (UID: \"b535e368-228c-49c3-b014-445acbad9d0d\") "
Nov 01 05:23:59.733416 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:59.733390    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b535e368-228c-49c3-b014-445acbad9d0d-openshift-service-ca" pod "b535e368-228c-49c3-b014-445acbad9d0d" (UID: "b535e368-228c-49c3-b014-445acbad9d0d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:23:59.803491 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:59.798641    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b535e368-228c-49c3-b014-445acbad9d0d-maas-api-tls" pod "b535e368-228c-49c3-b014-445acbad9d0d" (UID: "b535e368-228c-49c3-b014-445acbad9d0d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 05:23:59.803920 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:59.803887    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b535e368-228c-49c3-b014-445acbad9d0d-metrics-certs" pod "b535e368-228c-49c3-b014-445acbad9d0d" (UID: "b535e368-228c-49c3-b014-445acbad9d0d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 05:23:59.848079 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:59.847952    5689 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b535e368-228c-49c3-b014-445acbad9d0d-metrics-certs\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:23:59.855689 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:59.855653    5689 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b535e368-228c-49c3-b014-445acbad9d0d-maas-api-tls\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:23:59.855926 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:59.855908    5689 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b535e368-228c-49c3-b014-445acbad9d0d-openshift-service-ca\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:23:59.913479 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:59.903777    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b535e368-228c-49c3-b014-445acbad9d0d-kube-api-access-tlx8k" pod "b535e368-228c-49c3-b014-445acbad9d0d" (UID: "b535e368-228c-49c3-b014-445acbad9d0d"). InnerVolumeSpecName "kube-api-access-tlx8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:23:59.959085 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:23:59.957745    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlx8k\" (UniqueName: \"kubernetes.io/projected/b535e368-228c-49c3-b014-445acbad9d0d-kube-api-access-tlx8k\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:24:00.191922 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:00.191892    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af8051-8f8c87946-qzrqg"
Nov 01 05:24:00.196363 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:00.196331    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af8051-8f8c87946-qzrqg" event={"ID":"b535e368-228c-49c3-b014-445acbad9d0d","Type":"ContainerDied","Data":"0e464f3668bedb6d0def04d35c2dd4f98524ff363ce025a30ebdd047d9c81dc3"}
Nov 01 05:24:00.207576 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:00.207554    5689 scope.go:122] "RemoveContainer" containerID="5b4eac7be11fda86a4ac565c99e868b23848fbd275883a42eeed09b577f059cf"
Nov 01 05:24:00.370345 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:00.370309    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-af8051-8f8c87946-qzrqg"]
Nov 01 05:24:00.375932 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:00.374497    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-af8051-8f8c87946-qzrqg"]
Nov 01 05:24:00.487488 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:00.487289    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:24:00.534389 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:24:00.534327    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:24:00.985477 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:00.982596    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:24:02.086567 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:02.086534    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b535e368-228c-49c3-b014-445acbad9d0d" path="/var/lib/kubelet/pods/b535e368-228c-49c3-b014-445acbad9d0d/volumes"
Nov 01 05:24:04.057366 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:04.057334    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:24:04.234916 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:04.234887    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:24:04.670711 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:04.670667    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-af8051-5599f954b9-g8vp8" podUID="bc3118cb-d425-4294-a46c-12a554343918" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 05:24:05.676376 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:05.676348    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:24:06.163463 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:06.163435    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:24:06.336706 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:06.336668    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:24:07.127539 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:07.127503    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:24:07.307117 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:07.307082    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:24:07.716761 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:07.716727    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:24:08.347272 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:08.347207    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:24:08.638304 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:08.638230    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:24:10.189283 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:10.189247    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:24:10.668045 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:10.668016    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:24:11.746614 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:11.746581    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5451cb-6cff848b7f-f7psh"]
Nov 01 05:24:11.747133 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:11.747119    5689 cpu_manager.go:414] "containerMap: removing container" podUID="b535e368-228c-49c3-b014-445acbad9d0d" containerName="maas-api"
Nov 01 05:24:11.747176 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:11.747134    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b535e368-228c-49c3-b014-445acbad9d0d" containerName="maas-api"
Nov 01 05:24:11.747176 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:11.747146    5689 cpu_manager.go:414] "containerMap: removing container" podUID="731cd5ef-5a2a-4771-8067-75b133315fbf" containerName="payload-processing"
Nov 01 05:24:11.747176 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:11.747151    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="731cd5ef-5a2a-4771-8067-75b133315fbf" containerName="payload-processing"
Nov 01 05:24:11.747176 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:11.747168    5689 cpu_manager.go:414] "containerMap: removing container" podUID="61f28ebd-0530-4058-9f4b-a3b17b6c04aa" containerName="payload-pre-processing"
Nov 01 05:24:11.747176 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:11.747174    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61f28ebd-0530-4058-9f4b-a3b17b6c04aa" containerName="payload-pre-processing"
Nov 01 05:24:11.747327 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:11.747240    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="b535e368-228c-49c3-b014-445acbad9d0d" containerName="maas-api"
Nov 01 05:24:11.747327 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:11.747248    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="731cd5ef-5a2a-4771-8067-75b133315fbf" containerName="payload-processing"
Nov 01 05:24:11.747327 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:11.747256    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="61f28ebd-0530-4058-9f4b-a3b17b6c04aa" containerName="payload-pre-processing"
Nov 01 05:24:11.752288 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:11.752265    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5451cb-6cff848b7f-f7psh"
Nov 01 05:24:11.808920 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:11.808886    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5451cb-6cff848b7f-f7psh"]
Nov 01 05:24:11.855369 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:11.855327    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mswds\" (UniqueName: \"kubernetes.io/projected/f4f05151-c697-49d3-bc36-f3e071865802-kube-api-access-mswds\") pod \"payload-pre-processing-e2e-ipp-b-5451cb-6cff848b7f-f7psh\" (UID: \"f4f05151-c697-49d3-bc36-f3e071865802\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5451cb-6cff848b7f-f7psh"
Nov 01 05:24:11.855579 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:11.855500    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-5451cb\""
Nov 01 05:24:11.855579 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:11.855535    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-5451cb-dockercfg-wfxb9\""
Nov 01 05:24:11.855668 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:11.855599    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4f05151-c697-49d3-bc36-f3e071865802-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5451cb-6cff848b7f-f7psh\" (UID: \"f4f05151-c697-49d3-bc36-f3e071865802\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5451cb-6cff848b7f-f7psh"
Nov 01 05:24:11.957300 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:11.957267    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mswds\" (UniqueName: \"kubernetes.io/projected/f4f05151-c697-49d3-bc36-f3e071865802-kube-api-access-mswds\") pod \"payload-pre-processing-e2e-ipp-b-5451cb-6cff848b7f-f7psh\" (UID: \"f4f05151-c697-49d3-bc36-f3e071865802\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5451cb-6cff848b7f-f7psh"
Nov 01 05:24:11.957475 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:11.957423    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4f05151-c697-49d3-bc36-f3e071865802-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5451cb-6cff848b7f-f7psh\" (UID: \"f4f05151-c697-49d3-bc36-f3e071865802\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5451cb-6cff848b7f-f7psh"
Nov 01 05:24:11.958114 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:11.958094    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4f05151-c697-49d3-bc36-f3e071865802-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5451cb-6cff848b7f-f7psh\" (UID: \"f4f05151-c697-49d3-bc36-f3e071865802\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5451cb-6cff848b7f-f7psh"
Nov 01 05:24:12.017925 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:12.017857    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mswds\" (UniqueName: \"kubernetes.io/projected/f4f05151-c697-49d3-bc36-f3e071865802-kube-api-access-mswds\") pod \"payload-pre-processing-e2e-ipp-b-5451cb-6cff848b7f-f7psh\" (UID: \"f4f05151-c697-49d3-bc36-f3e071865802\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5451cb-6cff848b7f-f7psh"
Nov 01 05:24:12.066436 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:12.066406    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5451cb-6cff848b7f-f7psh"
Nov 01 05:24:12.363440 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:12.363363    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:24:12.411362 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:12.411330    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:24:12.443221 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:12.443175    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5451cb-6cff848b7f-f7psh"]
Nov 01 05:24:12.449225 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:24:12.449086    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4f05151_c697_49d3_bc36_f3e071865802.slice/crio-4ccd9dd84b75367f2c39c5484583b6ef69ad2b9d9e1bc8665afc072783e5dfdd WatchSource:0}: Error finding container 4ccd9dd84b75367f2c39c5484583b6ef69ad2b9d9e1bc8665afc072783e5dfdd: Status 404 returned error can't find the container with id 4ccd9dd84b75367f2c39c5484583b6ef69ad2b9d9e1bc8665afc072783e5dfdd
Nov 01 05:24:12.500912 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:12.500873    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5451cb-fdbf7d7f6-qwfh5"]
Nov 01 05:24:12.507056 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:12.507036    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5451cb-fdbf7d7f6-qwfh5"
Nov 01 05:24:12.542790 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:12.542761    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:24:12.566495 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:12.565756    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdb18db8-1611-478c-a170-26c41b29f020-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5451cb-fdbf7d7f6-qwfh5\" (UID: \"cdb18db8-1611-478c-a170-26c41b29f020\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5451cb-fdbf7d7f6-qwfh5"
Nov 01 05:24:12.566495 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:12.565867    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9jqk\" (UniqueName: \"kubernetes.io/projected/cdb18db8-1611-478c-a170-26c41b29f020-kube-api-access-k9jqk\") pod \"payload-processing-e2e-ipp-b-5451cb-fdbf7d7f6-qwfh5\" (UID: \"cdb18db8-1611-478c-a170-26c41b29f020\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5451cb-fdbf7d7f6-qwfh5"
Nov 01 05:24:12.570050 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:12.568739    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5451cb-fdbf7d7f6-qwfh5"]
Nov 01 05:24:12.682525 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:12.671214    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdb18db8-1611-478c-a170-26c41b29f020-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5451cb-fdbf7d7f6-qwfh5\" (UID: \"cdb18db8-1611-478c-a170-26c41b29f020\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5451cb-fdbf7d7f6-qwfh5"
Nov 01 05:24:12.682525 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:12.671328    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9jqk\" (UniqueName: \"kubernetes.io/projected/cdb18db8-1611-478c-a170-26c41b29f020-kube-api-access-k9jqk\") pod \"payload-processing-e2e-ipp-b-5451cb-fdbf7d7f6-qwfh5\" (UID: \"cdb18db8-1611-478c-a170-26c41b29f020\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5451cb-fdbf7d7f6-qwfh5"
Nov 01 05:24:12.762530 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:12.745682    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdb18db8-1611-478c-a170-26c41b29f020-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5451cb-fdbf7d7f6-qwfh5\" (UID: \"cdb18db8-1611-478c-a170-26c41b29f020\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5451cb-fdbf7d7f6-qwfh5"
Nov 01 05:24:12.794472 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:12.791664    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9jqk\" (UniqueName: \"kubernetes.io/projected/cdb18db8-1611-478c-a170-26c41b29f020-kube-api-access-k9jqk\") pod \"payload-processing-e2e-ipp-b-5451cb-fdbf7d7f6-qwfh5\" (UID: \"cdb18db8-1611-478c-a170-26c41b29f020\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5451cb-fdbf7d7f6-qwfh5"
Nov 01 05:24:12.816558 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:12.816521    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:24:12.839717 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:12.839684    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5451cb-fdbf7d7f6-qwfh5"
Nov 01 05:24:13.600534 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:13.595621    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5451cb-fdbf7d7f6-qwfh5"]
Nov 01 05:24:13.643480 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:13.631122    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5451cb-6cff848b7f-f7psh" event={"ID":"f4f05151-c697-49d3-bc36-f3e071865802","Type":"ContainerStarted","Data":"ebaf128572491b1eeb26c1265735a8f022f663f2ff86e27c286e5261f36470b8"}
Nov 01 05:24:13.643480 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:13.631191    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5451cb-6cff848b7f-f7psh" event={"ID":"f4f05151-c697-49d3-bc36-f3e071865802","Type":"ContainerStarted","Data":"4ccd9dd84b75367f2c39c5484583b6ef69ad2b9d9e1bc8665afc072783e5dfdd"}
Nov 01 05:24:13.643480 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:13.631820    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5451cb-6cff848b7f-f7psh"
Nov 01 05:24:13.740476 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:13.728783    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:24:13.870523 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:13.821061    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5451cb-6cff848b7f-f7psh" podStartSLOduration=2.821044629 podStartE2EDuration="2.821044629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:24:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:24:13.820231707 +0000 UTC m=+1231.891028422" watchObservedRunningTime="2026-11-01 05:24:13.821044629 +0000 UTC m=+1231.891841350"
Nov 01 05:24:14.528478 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:14.528002    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3d70ff/simulator-w2-3d70ff-kserve-898c5fb4f-t5bzk" podUID="05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 05:24:14.528478 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:14.528336    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a48099/simulator-w4-a48099-kserve-7df86f7d-rdlrm" podUID="12c3ad65-a804-429f-ab42-9cc7b2433855" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 01 05:24:14.672529 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:14.672489    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5451cb-fdbf7d7f6-qwfh5" event={"ID":"cdb18db8-1611-478c-a170-26c41b29f020","Type":"ContainerStarted","Data":"5272f36dc024b9ce2913b945d7e43d012eae0a91087738f0910e26611c587382"}
Nov 01 05:24:14.672789 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:14.672766    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5451cb-fdbf7d7f6-qwfh5" event={"ID":"cdb18db8-1611-478c-a170-26c41b29f020","Type":"ContainerStarted","Data":"c7323caec5569925c014cfa0f0d1c6b5ca17025a0f82cb4ef43175cbd48586c3"}
Nov 01 05:24:14.673005 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:14.672947    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-5451cb-fdbf7d7f6-qwfh5"
Nov 01 05:24:14.691575 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:14.691540    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-af8051-5599f954b9-g8vp8" podUID="bc3118cb-d425-4294-a46c-12a554343918" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 05:24:14.812543 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:14.798611    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-5451cb-fdbf7d7f6-qwfh5" podStartSLOduration=2.798589956 podStartE2EDuration="2.798589956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:24:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:24:14.795987134 +0000 UTC m=+1232.866783854" watchObservedRunningTime="2026-11-01 05:24:14.798589956 +0000 UTC m=+1232.869386678"
Nov 01 05:24:15.585477 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:15.585055    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a48099/simulator-w4-a48099-kserve-7df86f7d-rdlrm" podUID="12c3ad65-a804-429f-ab42-9cc7b2433855" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 05:24:15.585477 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:15.585435    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-a48099/simulator-w4-a48099-kserve-7df86f7d-rdlrm"
Nov 01 05:24:15.617998 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:15.617968    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-a48099/simulator-w4-a48099-kserve-7df86f7d-rdlrm"
Nov 01 05:24:15.970671 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:15.970640    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:24:16.151932 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:16.151893    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:24:18.635061 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:18.635040    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-af8051-5599f954b9-g8vp8"
Nov 01 05:24:18.660202 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:18.660176    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc3118cb-d425-4294-a46c-12a554343918-kube-api-access-2t2vx\" (UniqueName: \"kubernetes.io/projected/bc3118cb-d425-4294-a46c-12a554343918-kube-api-access-2t2vx\") pod \"bc3118cb-d425-4294-a46c-12a554343918\" (UID: \"bc3118cb-d425-4294-a46c-12a554343918\") "
Nov 01 05:24:18.660318 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:18.660220    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc3118cb-d425-4294-a46c-12a554343918-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc3118cb-d425-4294-a46c-12a554343918-plugins-config-volume\") pod \"bc3118cb-d425-4294-a46c-12a554343918\" (UID: \"bc3118cb-d425-4294-a46c-12a554343918\") "
Nov 01 05:24:18.660648 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:18.660625    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc3118cb-d425-4294-a46c-12a554343918-plugins-config-volume" pod "bc3118cb-d425-4294-a46c-12a554343918" (UID: "bc3118cb-d425-4294-a46c-12a554343918"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:24:18.662404 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:18.662387    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc3118cb-d425-4294-a46c-12a554343918-kube-api-access-2t2vx" pod "bc3118cb-d425-4294-a46c-12a554343918" (UID: "bc3118cb-d425-4294-a46c-12a554343918"). InnerVolumeSpecName "kube-api-access-2t2vx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:24:18.756064 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:18.756033    5689 generic.go:356] "Generic (PLEG): container finished" podID="bc3118cb-d425-4294-a46c-12a554343918" containerID="8b6e6819a781276f924c22ea6a6f6a39e4fbee54ea16d58a4dbb585c5d4a16f2" exitCode=0
Nov 01 05:24:18.756194 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:18.756080    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-af8051-5599f954b9-g8vp8"
Nov 01 05:24:18.756194 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:18.756112    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-af8051-5599f954b9-g8vp8" event={"ID":"bc3118cb-d425-4294-a46c-12a554343918","Type":"ContainerDied","Data":"8b6e6819a781276f924c22ea6a6f6a39e4fbee54ea16d58a4dbb585c5d4a16f2"}
Nov 01 05:24:18.756194 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:18.756164    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-af8051-5599f954b9-g8vp8" event={"ID":"bc3118cb-d425-4294-a46c-12a554343918","Type":"ContainerDied","Data":"690e934e6ca73de734172cad2cd3cb21be17ae572c7cc68adc351fc2acb80840"}
Nov 01 05:24:18.756194 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:18.756190    5689 scope.go:122] "RemoveContainer" containerID="8b6e6819a781276f924c22ea6a6f6a39e4fbee54ea16d58a4dbb585c5d4a16f2"
Nov 01 05:24:18.761410 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:18.761384    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2t2vx\" (UniqueName: \"kubernetes.io/projected/bc3118cb-d425-4294-a46c-12a554343918-kube-api-access-2t2vx\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:24:18.761410 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:18.761407    5689 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc3118cb-d425-4294-a46c-12a554343918-plugins-config-volume\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:24:18.766029 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:18.766013    5689 scope.go:122] "RemoveContainer" containerID="8b6e6819a781276f924c22ea6a6f6a39e4fbee54ea16d58a4dbb585c5d4a16f2"
Nov 01 05:24:18.766293 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:24:18.766268    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b6e6819a781276f924c22ea6a6f6a39e4fbee54ea16d58a4dbb585c5d4a16f2\": container with ID starting with 8b6e6819a781276f924c22ea6a6f6a39e4fbee54ea16d58a4dbb585c5d4a16f2 not found: ID does not exist" containerID="8b6e6819a781276f924c22ea6a6f6a39e4fbee54ea16d58a4dbb585c5d4a16f2"
Nov 01 05:24:18.766355 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:18.766303    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b6e6819a781276f924c22ea6a6f6a39e4fbee54ea16d58a4dbb585c5d4a16f2"} err="failed to get container status \"8b6e6819a781276f924c22ea6a6f6a39e4fbee54ea16d58a4dbb585c5d4a16f2\": rpc error: code = NotFound desc = could not find container \"8b6e6819a781276f924c22ea6a6f6a39e4fbee54ea16d58a4dbb585c5d4a16f2\": container with ID starting with 8b6e6819a781276f924c22ea6a6f6a39e4fbee54ea16d58a4dbb585c5d4a16f2 not found: ID does not exist"
Nov 01 05:24:18.919884 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:18.919856    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-af8051-5599f954b9-g8vp8"]
Nov 01 05:24:18.927212 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:18.927184    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-af8051-5599f954b9-g8vp8"]
Nov 01 05:24:19.219640 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:19.219556    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:24:20.068945 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:20.068912    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc3118cb-d425-4294-a46c-12a554343918" path="/var/lib/kubelet/pods/bc3118cb-d425-4294-a46c-12a554343918/volumes"
Nov 01 05:24:23.341996 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:23.341958    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:24:24.055172 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:24.053636    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:24:25.622605 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:25.622547    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a48099/simulator-w4-a48099-kserve-7df86f7d-rdlrm" podUID="12c3ad65-a804-429f-ab42-9cc7b2433855" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 05:24:25.755492 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:25.747611    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5451cb-6cff848b7f-f7psh"
Nov 01 05:24:26.746670 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:26.746635    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-5451cb-fdbf7d7f6-qwfh5"
Nov 01 05:24:29.114202 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:29.114170    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:24:29.311200 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:29.311161    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:24:30.069066 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:24:30.069021    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:24:33.900977 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:33.900927    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:24:38.839520 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:38.839428    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:24:40.242000 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:40.241967    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:24:45.519799 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:45.519762    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:24:50.475334 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:50.475239    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:24:56.747495 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:56.747442    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:24:56.929352 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:56.929317    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:24:57.111416 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:57.111335    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:24:57.473950 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:57.473918    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:24:57.809466 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:24:57.809375    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:25:00.068233 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:25:00.068193    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:25:00.844476 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:00.843501    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:25:00.899477 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:00.898739    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4s7z6"]
Nov 01 05:25:00.908944 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:00.908915    5689 cpu_manager.go:414] "containerMap: removing container" podUID="bc3118cb-d425-4294-a46c-12a554343918" containerName="payload-processing"
Nov 01 05:25:00.909160 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:00.909141    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc3118cb-d425-4294-a46c-12a554343918" containerName="payload-processing"
Nov 01 05:25:00.909438 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:00.909419    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc3118cb-d425-4294-a46c-12a554343918" containerName="payload-processing"
Nov 01 05:25:00.925532 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:00.918691    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4s7z6"
Nov 01 05:25:00.971493 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:00.971435    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5ljv7"]
Nov 01 05:25:00.972298 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:00.972266    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b68feecd-350f-4582-8cca-c966f065e929-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-4s7z6\" (UID: \"b68feecd-350f-4582-8cca-c966f065e929\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4s7z6"
Nov 01 05:25:00.972521 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:00.972497    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b68feecd-350f-4582-8cca-c966f065e929-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-4s7z6\" (UID: \"b68feecd-350f-4582-8cca-c966f065e929\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4s7z6"
Nov 01 05:25:00.972683 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:00.972664    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw55p\" (UniqueName: \"kubernetes.io/projected/b68feecd-350f-4582-8cca-c966f065e929-kube-api-access-hw55p\") pod \"maas-default-gateway-openshift-default-58d898bd55-4s7z6\" (UID: \"b68feecd-350f-4582-8cca-c966f065e929\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4s7z6"
Nov 01 05:25:00.972843 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:00.972827    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b68feecd-350f-4582-8cca-c966f065e929-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-4s7z6\" (UID: \"b68feecd-350f-4582-8cca-c966f065e929\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4s7z6"
Nov 01 05:25:00.973168 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:00.973146    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b68feecd-350f-4582-8cca-c966f065e929-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-4s7z6\" (UID: \"b68feecd-350f-4582-8cca-c966f065e929\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4s7z6"
Nov 01 05:25:00.973298 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:00.973282    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b68feecd-350f-4582-8cca-c966f065e929-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-4s7z6\" (UID: \"b68feecd-350f-4582-8cca-c966f065e929\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4s7z6"
Nov 01 05:25:00.973471 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:00.973436    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b68feecd-350f-4582-8cca-c966f065e929-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-4s7z6\" (UID: \"b68feecd-350f-4582-8cca-c966f065e929\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4s7z6"
Nov 01 05:25:00.973622 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:00.973604    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b68feecd-350f-4582-8cca-c966f065e929-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-4s7z6\" (UID: \"b68feecd-350f-4582-8cca-c966f065e929\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4s7z6"
Nov 01 05:25:00.973794 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:00.973777    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b68feecd-350f-4582-8cca-c966f065e929-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-4s7z6\" (UID: \"b68feecd-350f-4582-8cca-c966f065e929\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4s7z6"
Nov 01 05:25:00.996843 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:00.996648    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-8hgdn\""
Nov 01 05:25:01.011496 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:00.997571    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-z24p2"]
Nov 01 05:25:01.015482 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.012567    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5ljv7"
Nov 01 05:25:01.030485 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.028867    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4s7z6"]
Nov 01 05:25:01.030485 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.028978    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-z24p2"
Nov 01 05:25:01.067580 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.062343    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5ljv7"]
Nov 01 05:25:01.110579 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.103225    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b68feecd-350f-4582-8cca-c966f065e929-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-4s7z6\" (UID: \"b68feecd-350f-4582-8cca-c966f065e929\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4s7z6"
Nov 01 05:25:01.110579 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.103279    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b68feecd-350f-4582-8cca-c966f065e929-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-4s7z6\" (UID: \"b68feecd-350f-4582-8cca-c966f065e929\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4s7z6"
Nov 01 05:25:01.110579 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.103342    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b68feecd-350f-4582-8cca-c966f065e929-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-4s7z6\" (UID: \"b68feecd-350f-4582-8cca-c966f065e929\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4s7z6"
Nov 01 05:25:01.110579 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.103371    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b68feecd-350f-4582-8cca-c966f065e929-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-4s7z6\" (UID: \"b68feecd-350f-4582-8cca-c966f065e929\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4s7z6"
Nov 01 05:25:01.110579 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.103398    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b68feecd-350f-4582-8cca-c966f065e929-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-4s7z6\" (UID: \"b68feecd-350f-4582-8cca-c966f065e929\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4s7z6"
Nov 01 05:25:01.110579 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.103487    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b68feecd-350f-4582-8cca-c966f065e929-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-4s7z6\" (UID: \"b68feecd-350f-4582-8cca-c966f065e929\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4s7z6"
Nov 01 05:25:01.110579 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.103517    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b68feecd-350f-4582-8cca-c966f065e929-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-4s7z6\" (UID: \"b68feecd-350f-4582-8cca-c966f065e929\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4s7z6"
Nov 01 05:25:01.110579 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.103555    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw55p\" (UniqueName: \"kubernetes.io/projected/b68feecd-350f-4582-8cca-c966f065e929-kube-api-access-hw55p\") pod \"maas-default-gateway-openshift-default-58d898bd55-4s7z6\" (UID: \"b68feecd-350f-4582-8cca-c966f065e929\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4s7z6"
Nov 01 05:25:01.110579 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.103614    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b68feecd-350f-4582-8cca-c966f065e929-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-4s7z6\" (UID: \"b68feecd-350f-4582-8cca-c966f065e929\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4s7z6"
Nov 01 05:25:01.110579 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.104099    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b68feecd-350f-4582-8cca-c966f065e929-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-4s7z6\" (UID: \"b68feecd-350f-4582-8cca-c966f065e929\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4s7z6"
Nov 01 05:25:01.110579 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.104372    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b68feecd-350f-4582-8cca-c966f065e929-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-4s7z6\" (UID: \"b68feecd-350f-4582-8cca-c966f065e929\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4s7z6"
Nov 01 05:25:01.110579 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.104937    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b68feecd-350f-4582-8cca-c966f065e929-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-4s7z6\" (UID: \"b68feecd-350f-4582-8cca-c966f065e929\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4s7z6"
Nov 01 05:25:01.212478 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.116029    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b68feecd-350f-4582-8cca-c966f065e929-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-4s7z6\" (UID: \"b68feecd-350f-4582-8cca-c966f065e929\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4s7z6"
Nov 01 05:25:01.212766 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.129237    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b68feecd-350f-4582-8cca-c966f065e929-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-4s7z6\" (UID: \"b68feecd-350f-4582-8cca-c966f065e929\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4s7z6"
Nov 01 05:25:01.220476 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.179964    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b68feecd-350f-4582-8cca-c966f065e929-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-4s7z6\" (UID: \"b68feecd-350f-4582-8cca-c966f065e929\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4s7z6"
Nov 01 05:25:01.220662 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.208780    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b68feecd-350f-4582-8cca-c966f065e929-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-4s7z6\" (UID: \"b68feecd-350f-4582-8cca-c966f065e929\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4s7z6"
Nov 01 05:25:01.224520 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.209639    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad96da25-2a2b-471c-9a7c-2c3f166553de-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-z24p2\" (UID: \"ad96da25-2a2b-471c-9a7c-2c3f166553de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-z24p2"
Nov 01 05:25:01.255645 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.255610    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad96da25-2a2b-471c-9a7c-2c3f166553de-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-z24p2\" (UID: \"ad96da25-2a2b-471c-9a7c-2c3f166553de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-z24p2"
Nov 01 05:25:01.255917 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.255895    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad96da25-2a2b-471c-9a7c-2c3f166553de-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-z24p2\" (UID: \"ad96da25-2a2b-471c-9a7c-2c3f166553de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-z24p2"
Nov 01 05:25:01.256092 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.256071    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82632280-5edc-47e8-a570-f98de80de625-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-5ljv7\" (UID: \"82632280-5edc-47e8-a570-f98de80de625\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5ljv7"
Nov 01 05:25:01.262489 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.256248    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82632280-5edc-47e8-a570-f98de80de625-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-5ljv7\" (UID: \"82632280-5edc-47e8-a570-f98de80de625\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5ljv7"
Nov 01 05:25:01.262489 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.261607    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad96da25-2a2b-471c-9a7c-2c3f166553de-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-z24p2\" (UID: \"ad96da25-2a2b-471c-9a7c-2c3f166553de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-z24p2"
Nov 01 05:25:01.286478 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.282704    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rncn7\" (UniqueName: \"kubernetes.io/projected/82632280-5edc-47e8-a570-f98de80de625-kube-api-access-rncn7\") pod \"maas-default-gateway-openshift-default-58d898bd55-5ljv7\" (UID: \"82632280-5edc-47e8-a570-f98de80de625\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5ljv7"
Nov 01 05:25:01.286478 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.282774    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82632280-5edc-47e8-a570-f98de80de625-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-5ljv7\" (UID: \"82632280-5edc-47e8-a570-f98de80de625\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5ljv7"
Nov 01 05:25:01.286478 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.282808    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82632280-5edc-47e8-a570-f98de80de625-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-5ljv7\" (UID: \"82632280-5edc-47e8-a570-f98de80de625\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5ljv7"
Nov 01 05:25:01.286478 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.282843    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82632280-5edc-47e8-a570-f98de80de625-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-5ljv7\" (UID: \"82632280-5edc-47e8-a570-f98de80de625\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5ljv7"
Nov 01 05:25:01.286478 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.282875    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad96da25-2a2b-471c-9a7c-2c3f166553de-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-z24p2\" (UID: \"ad96da25-2a2b-471c-9a7c-2c3f166553de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-z24p2"
Nov 01 05:25:01.286478 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.282915    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad96da25-2a2b-471c-9a7c-2c3f166553de-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-z24p2\" (UID: \"ad96da25-2a2b-471c-9a7c-2c3f166553de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-z24p2"
Nov 01 05:25:01.286478 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.282992    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82632280-5edc-47e8-a570-f98de80de625-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-5ljv7\" (UID: \"82632280-5edc-47e8-a570-f98de80de625\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5ljv7"
Nov 01 05:25:01.286478 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.283036    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad96da25-2a2b-471c-9a7c-2c3f166553de-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-z24p2\" (UID: \"ad96da25-2a2b-471c-9a7c-2c3f166553de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-z24p2"
Nov 01 05:25:01.286478 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.283081    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82632280-5edc-47e8-a570-f98de80de625-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-5ljv7\" (UID: \"82632280-5edc-47e8-a570-f98de80de625\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5ljv7"
Nov 01 05:25:01.286478 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.283111    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82632280-5edc-47e8-a570-f98de80de625-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-5ljv7\" (UID: \"82632280-5edc-47e8-a570-f98de80de625\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5ljv7"
Nov 01 05:25:01.286478 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.283143    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf4wt\" (UniqueName: \"kubernetes.io/projected/ad96da25-2a2b-471c-9a7c-2c3f166553de-kube-api-access-gf4wt\") pod \"maas-default-gateway-openshift-default-58d898bd55-z24p2\" (UID: \"ad96da25-2a2b-471c-9a7c-2c3f166553de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-z24p2"
Nov 01 05:25:01.286478 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.283193    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad96da25-2a2b-471c-9a7c-2c3f166553de-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-z24p2\" (UID: \"ad96da25-2a2b-471c-9a7c-2c3f166553de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-z24p2"
Nov 01 05:25:01.333479 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.333035    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b68feecd-350f-4582-8cca-c966f065e929-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-4s7z6\" (UID: \"b68feecd-350f-4582-8cca-c966f065e929\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4s7z6"
Nov 01 05:25:01.354478 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.350078    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-z24p2"]
Nov 01 05:25:01.385483 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.383158    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:25:01.391874 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.387254    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad96da25-2a2b-471c-9a7c-2c3f166553de-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-z24p2\" (UID: \"ad96da25-2a2b-471c-9a7c-2c3f166553de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-z24p2"
Nov 01 05:25:01.391874 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.387339    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rncn7\" (UniqueName: \"kubernetes.io/projected/82632280-5edc-47e8-a570-f98de80de625-kube-api-access-rncn7\") pod \"maas-default-gateway-openshift-default-58d898bd55-5ljv7\" (UID: \"82632280-5edc-47e8-a570-f98de80de625\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5ljv7"
Nov 01 05:25:01.391874 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.387385    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82632280-5edc-47e8-a570-f98de80de625-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-5ljv7\" (UID: \"82632280-5edc-47e8-a570-f98de80de625\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5ljv7"
Nov 01 05:25:01.391874 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.387415    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82632280-5edc-47e8-a570-f98de80de625-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-5ljv7\" (UID: \"82632280-5edc-47e8-a570-f98de80de625\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5ljv7"
Nov 01 05:25:01.391874 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.387469    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82632280-5edc-47e8-a570-f98de80de625-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-5ljv7\" (UID: \"82632280-5edc-47e8-a570-f98de80de625\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5ljv7"
Nov 01 05:25:01.391874 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.387500    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad96da25-2a2b-471c-9a7c-2c3f166553de-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-z24p2\" (UID: \"ad96da25-2a2b-471c-9a7c-2c3f166553de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-z24p2"
Nov 01 05:25:01.391874 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.387536    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad96da25-2a2b-471c-9a7c-2c3f166553de-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-z24p2\" (UID: \"ad96da25-2a2b-471c-9a7c-2c3f166553de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-z24p2"
Nov 01 05:25:01.391874 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.387600    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82632280-5edc-47e8-a570-f98de80de625-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-5ljv7\" (UID: \"82632280-5edc-47e8-a570-f98de80de625\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5ljv7"
Nov 01 05:25:01.391874 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.387639    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad96da25-2a2b-471c-9a7c-2c3f166553de-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-z24p2\" (UID: \"ad96da25-2a2b-471c-9a7c-2c3f166553de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-z24p2"
Nov 01 05:25:01.391874 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.387677    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82632280-5edc-47e8-a570-f98de80de625-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-5ljv7\" (UID: \"82632280-5edc-47e8-a570-f98de80de625\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5ljv7"
Nov 01 05:25:01.391874 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.387708    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82632280-5edc-47e8-a570-f98de80de625-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-5ljv7\" (UID: \"82632280-5edc-47e8-a570-f98de80de625\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5ljv7"
Nov 01 05:25:01.391874 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.387740    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf4wt\" (UniqueName: \"kubernetes.io/projected/ad96da25-2a2b-471c-9a7c-2c3f166553de-kube-api-access-gf4wt\") pod \"maas-default-gateway-openshift-default-58d898bd55-z24p2\" (UID: \"ad96da25-2a2b-471c-9a7c-2c3f166553de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-z24p2"
Nov 01 05:25:01.391874 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.387782    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad96da25-2a2b-471c-9a7c-2c3f166553de-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-z24p2\" (UID: \"ad96da25-2a2b-471c-9a7c-2c3f166553de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-z24p2"
Nov 01 05:25:01.391874 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.387861    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad96da25-2a2b-471c-9a7c-2c3f166553de-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-z24p2\" (UID: \"ad96da25-2a2b-471c-9a7c-2c3f166553de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-z24p2"
Nov 01 05:25:01.391874 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.387891    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad96da25-2a2b-471c-9a7c-2c3f166553de-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-z24p2\" (UID: \"ad96da25-2a2b-471c-9a7c-2c3f166553de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-z24p2"
Nov 01 05:25:01.391874 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.387929    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad96da25-2a2b-471c-9a7c-2c3f166553de-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-z24p2\" (UID: \"ad96da25-2a2b-471c-9a7c-2c3f166553de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-z24p2"
Nov 01 05:25:01.391874 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.387984    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82632280-5edc-47e8-a570-f98de80de625-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-5ljv7\" (UID: \"82632280-5edc-47e8-a570-f98de80de625\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5ljv7"
Nov 01 05:25:01.391874 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.388009    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82632280-5edc-47e8-a570-f98de80de625-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-5ljv7\" (UID: \"82632280-5edc-47e8-a570-f98de80de625\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5ljv7"
Nov 01 05:25:01.391874 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.388036    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82632280-5edc-47e8-a570-f98de80de625-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-5ljv7\" (UID: \"82632280-5edc-47e8-a570-f98de80de625\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5ljv7"
Nov 01 05:25:01.391874 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.388343    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad96da25-2a2b-471c-9a7c-2c3f166553de-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-z24p2\" (UID: \"ad96da25-2a2b-471c-9a7c-2c3f166553de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-z24p2"
Nov 01 05:25:01.391874 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.388372    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82632280-5edc-47e8-a570-f98de80de625-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-5ljv7\" (UID: \"82632280-5edc-47e8-a570-f98de80de625\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5ljv7"
Nov 01 05:25:01.391874 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.388519    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82632280-5edc-47e8-a570-f98de80de625-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-5ljv7\" (UID: \"82632280-5edc-47e8-a570-f98de80de625\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5ljv7"
Nov 01 05:25:01.391874 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.388816    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82632280-5edc-47e8-a570-f98de80de625-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-5ljv7\" (UID: \"82632280-5edc-47e8-a570-f98de80de625\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5ljv7"
Nov 01 05:25:01.391874 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.389042    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad96da25-2a2b-471c-9a7c-2c3f166553de-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-z24p2\" (UID: \"ad96da25-2a2b-471c-9a7c-2c3f166553de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-z24p2"
Nov 01 05:25:01.391874 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.389089    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad96da25-2a2b-471c-9a7c-2c3f166553de-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-z24p2\" (UID: \"ad96da25-2a2b-471c-9a7c-2c3f166553de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-z24p2"
Nov 01 05:25:01.391874 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.389582    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad96da25-2a2b-471c-9a7c-2c3f166553de-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-z24p2\" (UID: \"ad96da25-2a2b-471c-9a7c-2c3f166553de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-z24p2"
Nov 01 05:25:01.391874 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.391561    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82632280-5edc-47e8-a570-f98de80de625-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-5ljv7\" (UID: \"82632280-5edc-47e8-a570-f98de80de625\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5ljv7"
Nov 01 05:25:01.391874 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.391811    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad96da25-2a2b-471c-9a7c-2c3f166553de-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-z24p2\" (UID: \"ad96da25-2a2b-471c-9a7c-2c3f166553de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-z24p2"
Nov 01 05:25:01.396143 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.395416    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad96da25-2a2b-471c-9a7c-2c3f166553de-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-z24p2\" (UID: \"ad96da25-2a2b-471c-9a7c-2c3f166553de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-z24p2"
Nov 01 05:25:01.411323 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.411292    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw55p\" (UniqueName: \"kubernetes.io/projected/b68feecd-350f-4582-8cca-c966f065e929-kube-api-access-hw55p\") pod \"maas-default-gateway-openshift-default-58d898bd55-4s7z6\" (UID: \"b68feecd-350f-4582-8cca-c966f065e929\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4s7z6"
Nov 01 05:25:01.413300 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.413252    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82632280-5edc-47e8-a570-f98de80de625-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-5ljv7\" (UID: \"82632280-5edc-47e8-a570-f98de80de625\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5ljv7"
Nov 01 05:25:01.413731 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.413678    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad96da25-2a2b-471c-9a7c-2c3f166553de-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-z24p2\" (UID: \"ad96da25-2a2b-471c-9a7c-2c3f166553de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-z24p2"
Nov 01 05:25:01.416153 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.416127    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82632280-5edc-47e8-a570-f98de80de625-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-5ljv7\" (UID: \"82632280-5edc-47e8-a570-f98de80de625\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5ljv7"
Nov 01 05:25:01.451075 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.451013    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rncn7\" (UniqueName: \"kubernetes.io/projected/82632280-5edc-47e8-a570-f98de80de625-kube-api-access-rncn7\") pod \"maas-default-gateway-openshift-default-58d898bd55-5ljv7\" (UID: \"82632280-5edc-47e8-a570-f98de80de625\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5ljv7"
Nov 01 05:25:01.451650 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.451595    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad96da25-2a2b-471c-9a7c-2c3f166553de-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-z24p2\" (UID: \"ad96da25-2a2b-471c-9a7c-2c3f166553de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-z24p2"
Nov 01 05:25:01.451877 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.451854    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82632280-5edc-47e8-a570-f98de80de625-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-5ljv7\" (UID: \"82632280-5edc-47e8-a570-f98de80de625\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5ljv7"
Nov 01 05:25:01.452400 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.452378    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf4wt\" (UniqueName: \"kubernetes.io/projected/ad96da25-2a2b-471c-9a7c-2c3f166553de-kube-api-access-gf4wt\") pod \"maas-default-gateway-openshift-default-58d898bd55-z24p2\" (UID: \"ad96da25-2a2b-471c-9a7c-2c3f166553de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-z24p2"
Nov 01 05:25:01.575009 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.574966    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4s7z6"
Nov 01 05:25:01.579510 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.579484    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:25:01.651581 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.651298    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5ljv7"
Nov 01 05:25:01.689202 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.688979    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-z24p2"
Nov 01 05:25:01.763173 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:01.763143    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:25:02.010118 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:02.010096    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4s7z6"]
Nov 01 05:25:02.012650 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:25:02.012622    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb68feecd_350f_4582_8cca_c966f065e929.slice/crio-bfe6aebabbda4f4c038d5e2492659c04c424a9b3d2eec279106fc13c7ea2ea6b WatchSource:0}: Error finding container bfe6aebabbda4f4c038d5e2492659c04c424a9b3d2eec279106fc13c7ea2ea6b: Status 404 returned error can't find the container with id bfe6aebabbda4f4c038d5e2492659c04c424a9b3d2eec279106fc13c7ea2ea6b
Nov 01 05:25:02.014988 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:02.014959    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:25:02.015054 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:02.015017    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:25:02.015054 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:02.015046    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:25:02.075538 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:02.075485    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:25:02.076713 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:02.076683    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:25:02.076830 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:02.076744    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:25:02.078929 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:02.078909    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5ljv7"]
Nov 01 05:25:02.113675 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:02.113652    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-z24p2"]
Nov 01 05:25:02.115703 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:02.115678    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:25:02.129912 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:02.126919    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4s7z6" event={"ID":"b68feecd-350f-4582-8cca-c966f065e929","Type":"ContainerStarted","Data":"2c6c8a5d3577de09ff6c5bffc2e59004fd58614bdd4422ee1314d4a62fc5c716"}
Nov 01 05:25:02.129912 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:02.126981    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4s7z6" event={"ID":"b68feecd-350f-4582-8cca-c966f065e929","Type":"ContainerStarted","Data":"bfe6aebabbda4f4c038d5e2492659c04c424a9b3d2eec279106fc13c7ea2ea6b"}
Nov 01 05:25:02.129912 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:02.129893    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5ljv7" event={"ID":"82632280-5edc-47e8-a570-f98de80de625","Type":"ContainerStarted","Data":"1136ad74033d8b4d6a2e9efdad5cb0098412b703988e6044c9b5eaf573881a1b"}
Nov 01 05:25:02.134335 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:25:02.130330    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad96da25_2a2b_471c_9a7c_2c3f166553de.slice/crio-03aad61553edc1db9440defc857ed4d2b463790abbc0428b2955e36b5ee8db89 WatchSource:0}: Error finding container 03aad61553edc1db9440defc857ed4d2b463790abbc0428b2955e36b5ee8db89: Status 404 returned error can't find the container with id 03aad61553edc1db9440defc857ed4d2b463790abbc0428b2955e36b5ee8db89
Nov 01 05:25:02.134335 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:02.134017    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:25:02.134335 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:02.134094    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:25:02.134335 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:02.134128    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:25:02.249533 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:02.249469    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4s7z6" podStartSLOduration=2.249437345 podStartE2EDuration="2.249437345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:25:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:25:02.248237146 +0000 UTC m=+1280.319033935" watchObservedRunningTime="2026-11-01 05:25:02.249437345 +0000 UTC m=+1280.320234061"
Nov 01 05:25:02.576053 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:02.576017    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4s7z6"
Nov 01 05:25:03.140213 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:03.140175    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5ljv7" event={"ID":"82632280-5edc-47e8-a570-f98de80de625","Type":"ContainerStarted","Data":"b10ceda72be8c8b582ad06854fd96170f7d7778826dff56bc054ecd22b160dc0"}
Nov 01 05:25:03.142042 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:03.142019    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-z24p2" event={"ID":"ad96da25-2a2b-471c-9a7c-2c3f166553de","Type":"ContainerStarted","Data":"3ffc60f3476870f8de51f34f26fc3c7a366ba32d34c26d5823d3be67307b7b19"}
Nov 01 05:25:03.142156 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:03.142062    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-z24p2" event={"ID":"ad96da25-2a2b-471c-9a7c-2c3f166553de","Type":"ContainerStarted","Data":"03aad61553edc1db9440defc857ed4d2b463790abbc0428b2955e36b5ee8db89"}
Nov 01 05:25:03.271335 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:03.271282    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5ljv7" podStartSLOduration=3.2712671220000002 podStartE2EDuration="3.271267122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:25:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:25:03.26808922 +0000 UTC m=+1281.338885939" watchObservedRunningTime="2026-11-01 05:25:03.271267122 +0000 UTC m=+1281.342063841"
Nov 01 05:25:03.389058 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:03.388994    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-z24p2" podStartSLOduration=3.388977054 podStartE2EDuration="3.388977054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:25:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:25:03.388070676 +0000 UTC m=+1281.458867396" watchObservedRunningTime="2026-11-01 05:25:03.388977054 +0000 UTC m=+1281.459773774"
Nov 01 05:25:03.594478 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:03.593572    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-4s7z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 01 05:25:03.594478 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:03.593632    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4s7z6" podUID="b68feecd-350f-4582-8cca-c966f065e929" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 01 05:25:03.652876 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:03.652837    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5ljv7"
Nov 01 05:25:03.654823 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:03.654794    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-5ljv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Nov 01 05:25:03.654971 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:03.654846    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5ljv7" podUID="82632280-5edc-47e8-a570-f98de80de625" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Nov 01 05:25:03.706304 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:03.702162    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-z24p2"
Nov 01 05:25:03.717491 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:03.709944    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-z24p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 01 05:25:03.717491 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:03.710002    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-z24p2" podUID="ad96da25-2a2b-471c-9a7c-2c3f166553de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 01 05:25:04.580480 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:04.578783    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-4s7z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 01 05:25:04.580480 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:04.578843    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4s7z6" podUID="b68feecd-350f-4582-8cca-c966f065e929" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 01 05:25:04.653527 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:04.652837    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-5ljv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Nov 01 05:25:04.653527 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:04.652888    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5ljv7" podUID="82632280-5edc-47e8-a570-f98de80de625" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Nov 01 05:25:04.693659 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:04.693610    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-z24p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 01 05:25:04.693987 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:04.693932    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-z24p2" podUID="ad96da25-2a2b-471c-9a7c-2c3f166553de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 01 05:25:05.233482 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:05.231034    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:25:05.422029 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:05.421955    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:25:05.604528 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:05.589283    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-4s7z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 01 05:25:05.604528 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:05.589343    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4s7z6" podUID="b68feecd-350f-4582-8cca-c966f065e929" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 01 05:25:05.666477 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:05.652921    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-5ljv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Nov 01 05:25:05.666477 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:05.652978    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5ljv7" podUID="82632280-5edc-47e8-a570-f98de80de625" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Nov 01 05:25:05.844480 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:05.831557    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-z24p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 01 05:25:05.844480 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:05.831619    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-z24p2" podUID="ad96da25-2a2b-471c-9a7c-2c3f166553de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 01 05:25:06.583487 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:06.583428    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-4s7z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 01 05:25:06.583773 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:06.583744    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4s7z6" podUID="b68feecd-350f-4582-8cca-c966f065e929" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 01 05:25:06.652940 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:06.652909    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-5ljv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Nov 01 05:25:06.653407 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:06.653377    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5ljv7" podUID="82632280-5edc-47e8-a570-f98de80de625" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Nov 01 05:25:06.698732 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:06.698696    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-z24p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 01 05:25:06.698997 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:06.698964    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-z24p2" podUID="ad96da25-2a2b-471c-9a7c-2c3f166553de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 01 05:25:07.403049 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:07.402943    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:25:07.626442 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:07.626404    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-4s7z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 01 05:25:07.626761 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:07.626733    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4s7z6" podUID="b68feecd-350f-4582-8cca-c966f065e929" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 01 05:25:07.655560 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:07.653417    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-5ljv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Nov 01 05:25:07.656120 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:07.656084    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5ljv7" podUID="82632280-5edc-47e8-a570-f98de80de625" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Nov 01 05:25:07.695593 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:07.695556    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-z24p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 01 05:25:07.695863 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:07.695836    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-z24p2" podUID="ad96da25-2a2b-471c-9a7c-2c3f166553de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 01 05:25:08.576859 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:08.576824    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-4s7z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 01 05:25:08.577148 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:08.577120    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4s7z6" podUID="b68feecd-350f-4582-8cca-c966f065e929" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 01 05:25:08.655917 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:08.655883    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-5ljv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Nov 01 05:25:08.656378 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:08.656253    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5ljv7" podUID="82632280-5edc-47e8-a570-f98de80de625" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Nov 01 05:25:08.691037 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:08.690989    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-z24p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 01 05:25:08.691329 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:08.691288    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-z24p2" podUID="ad96da25-2a2b-471c-9a7c-2c3f166553de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 01 05:25:09.575832 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:09.575789    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-4s7z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 01 05:25:09.576007 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:09.575847    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4s7z6" podUID="b68feecd-350f-4582-8cca-c966f065e929" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 01 05:25:09.652230 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:09.652187    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-5ljv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Nov 01 05:25:09.652389 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:09.652250    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5ljv7" podUID="82632280-5edc-47e8-a570-f98de80de625" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Nov 01 05:25:09.689283 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:09.689252    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-z24p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 01 05:25:09.689684 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:09.689308    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-z24p2" podUID="ad96da25-2a2b-471c-9a7c-2c3f166553de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 01 05:25:10.575743 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:10.575688    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-4s7z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 01 05:25:10.576051 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:10.576019    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4s7z6" podUID="b68feecd-350f-4582-8cca-c966f065e929" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 01 05:25:11.692637 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:11.692587    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-5ljv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 05:25:11.693133 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:11.692647    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5ljv7" podUID="82632280-5edc-47e8-a570-f98de80de625" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 05:25:11.710070 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:11.709578    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-z24p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 05:25:11.710070 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:11.709639    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-z24p2" podUID="ad96da25-2a2b-471c-9a7c-2c3f166553de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 05:25:11.785766 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:11.785735    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-z24p2"
Nov 01 05:25:11.786740 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:11.786715    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-z24p2"
Nov 01 05:25:11.787748 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:11.787725    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-z24p2"
Nov 01 05:25:11.800933 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:11.800912    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5ljv7"
Nov 01 05:25:11.802516 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:11.802499    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5ljv7"
Nov 01 05:25:11.803735 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:11.803692    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5ljv7"
Nov 01 05:25:11.932011 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:11.931981    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4s7z6"
Nov 01 05:25:11.932533 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:11.932505    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4s7z6"
Nov 01 05:25:11.933254 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:11.933236    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4s7z6"
Nov 01 05:25:12.221401 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:12.221374    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:25:12.733215 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:12.733183    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:25:12.908668 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:12.908626    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:25:16.983544 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:16.983510    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:25:17.041320 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:17.041291    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:25:18.089538 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:18.089500    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:25:21.859446 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:21.859415    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:25:24.271307 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:24.271277    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:25:25.718947 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:25.718913    5689 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-2864b6-6d848878b6-zcftx_390739de-f035-41a9-b2d9-ff8664766ba6/payload-processing/0.log"
Nov 01 05:25:26.548732 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:26.548702    5689 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-5451cb-fdbf7d7f6-qwfh5_cdb18db8-1611-478c-a170-26c41b29f020/payload-processing/0.log"
Nov 01 05:25:26.762746 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:26.762717    5689 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-5451cb-fdbf7d7f6-qwfh5_cdb18db8-1611-478c-a170-26c41b29f020/payload-processing/0.log"
Nov 01 05:25:26.822193 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:26.822125    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:25:27.684243 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:27.684174    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:25:27.869770 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:27.869741    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:25:30.074180 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:25:30.074136    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:25:31.668228 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:31.668194    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:25:32.866969 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:32.866931    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:25:33.350107 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:33.350074    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:25:36.369572 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:36.369541    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:25:36.572230 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:36.572199    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:25:38.316809 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:38.316777    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:25:39.820865 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:39.820836    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:25:42.525693 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:42.525662    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:25:44.920622 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:44.920586    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:25:45.945367 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:45.945340    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:25:46.130076 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:46.130040    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:25:48.389993 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:48.389951    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:25:49.734666 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:49.734623    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:25:49.927491 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:49.917169    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:25:50.163768 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:50.163740    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:25:50.832354 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:50.832319    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:25:51.031116 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:51.031079    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:25:54.390090 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:54.390044    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:25:55.404600 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:55.404561    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:25:55.584844 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:55.584799    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:25:56.224970 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:56.224929    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:25:56.417532 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:56.417495    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:25:56.612139 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:25:56.612035    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:26:00.069826 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:26:00.069793    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:26:00.302338 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:00.302308    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:26:00.815372 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:00.815324    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:26:01.057744 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:01.057689    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:26:01.245715 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:01.245680    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:26:01.432670 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:01.432642    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:26:02.390047 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:02.390011    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:26:02.580048 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:02.579957    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:26:04.288179 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:04.288145    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:26:04.983663 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:04.983631    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:26:05.162012 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:05.161964    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:26:05.341652 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:05.341550    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:26:05.662477 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:05.655146    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:26:07.660288 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:07.660249    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:26:07.877787 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:07.877587    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:26:08.063802 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:08.063714    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ca7a55-747b56569f-5s7gh"]
Nov 01 05:26:08.068986 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:08.068960    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ca7a55-747b56569f-5s7gh"
Nov 01 05:26:08.094660 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:08.094636    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwxwz\" (UniqueName: \"kubernetes.io/projected/331a126d-11b4-4e43-88c3-5f2459793348-kube-api-access-gwxwz\") pod \"payload-pre-processing-e2e-praxis-b-ca7a55-747b56569f-5s7gh\" (UID: \"331a126d-11b4-4e43-88c3-5f2459793348\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ca7a55-747b56569f-5s7gh"
Nov 01 05:26:08.094776 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:08.094760    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/331a126d-11b4-4e43-88c3-5f2459793348-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-ca7a55-747b56569f-5s7gh\" (UID: \"331a126d-11b4-4e43-88c3-5f2459793348\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ca7a55-747b56569f-5s7gh"
Nov 01 05:26:08.120138 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:08.120112    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ca7a55-747b56569f-5s7gh"]
Nov 01 05:26:08.168949 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:08.168819    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-ca7a55-dockercfg-xhll8\""
Nov 01 05:26:08.168949 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:08.168915    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-ca7a55\""
Nov 01 05:26:08.243992 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:08.243845    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/331a126d-11b4-4e43-88c3-5f2459793348-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-ca7a55-747b56569f-5s7gh\" (UID: \"331a126d-11b4-4e43-88c3-5f2459793348\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ca7a55-747b56569f-5s7gh"
Nov 01 05:26:08.261338 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:08.261297    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwxwz\" (UniqueName: \"kubernetes.io/projected/331a126d-11b4-4e43-88c3-5f2459793348-kube-api-access-gwxwz\") pod \"payload-pre-processing-e2e-praxis-b-ca7a55-747b56569f-5s7gh\" (UID: \"331a126d-11b4-4e43-88c3-5f2459793348\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ca7a55-747b56569f-5s7gh"
Nov 01 05:26:08.264582 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:08.263118    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/331a126d-11b4-4e43-88c3-5f2459793348-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-ca7a55-747b56569f-5s7gh\" (UID: \"331a126d-11b4-4e43-88c3-5f2459793348\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ca7a55-747b56569f-5s7gh"
Nov 01 05:26:08.308460 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:08.308421    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ca7a55-74bd57dbd9-jrwrq"]
Nov 01 05:26:08.317533 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:08.316478    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-ca7a55-74bd57dbd9-jrwrq"
Nov 01 05:26:08.321508 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:08.320333    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ca7a55-74bd57dbd9-jrwrq"]
Nov 01 05:26:08.333149 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:08.333117    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwxwz\" (UniqueName: \"kubernetes.io/projected/331a126d-11b4-4e43-88c3-5f2459793348-kube-api-access-gwxwz\") pod \"payload-pre-processing-e2e-praxis-b-ca7a55-747b56569f-5s7gh\" (UID: \"331a126d-11b4-4e43-88c3-5f2459793348\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ca7a55-747b56569f-5s7gh"
Nov 01 05:26:08.382954 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:08.382927    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ca7a55-747b56569f-5s7gh"
Nov 01 05:26:08.469496 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:08.468815    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6wbq\" (UniqueName: \"kubernetes.io/projected/a984df19-49d0-4c90-a51d-b1028220ee61-kube-api-access-t6wbq\") pod \"payload-processing-e2e-praxis-b-ca7a55-74bd57dbd9-jrwrq\" (UID: \"a984df19-49d0-4c90-a51d-b1028220ee61\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ca7a55-74bd57dbd9-jrwrq"
Nov 01 05:26:08.469496 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:08.468865    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a984df19-49d0-4c90-a51d-b1028220ee61-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-ca7a55-74bd57dbd9-jrwrq\" (UID: \"a984df19-49d0-4c90-a51d-b1028220ee61\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ca7a55-74bd57dbd9-jrwrq"
Nov 01 05:26:08.570842 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:08.570757    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6wbq\" (UniqueName: \"kubernetes.io/projected/a984df19-49d0-4c90-a51d-b1028220ee61-kube-api-access-t6wbq\") pod \"payload-processing-e2e-praxis-b-ca7a55-74bd57dbd9-jrwrq\" (UID: \"a984df19-49d0-4c90-a51d-b1028220ee61\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ca7a55-74bd57dbd9-jrwrq"
Nov 01 05:26:08.570842 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:08.570794    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a984df19-49d0-4c90-a51d-b1028220ee61-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-ca7a55-74bd57dbd9-jrwrq\" (UID: \"a984df19-49d0-4c90-a51d-b1028220ee61\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ca7a55-74bd57dbd9-jrwrq"
Nov 01 05:26:08.571443 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:08.571425    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a984df19-49d0-4c90-a51d-b1028220ee61-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-ca7a55-74bd57dbd9-jrwrq\" (UID: \"a984df19-49d0-4c90-a51d-b1028220ee61\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ca7a55-74bd57dbd9-jrwrq"
Nov 01 05:26:08.631690 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:08.631664    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6wbq\" (UniqueName: \"kubernetes.io/projected/a984df19-49d0-4c90-a51d-b1028220ee61-kube-api-access-t6wbq\") pod \"payload-processing-e2e-praxis-b-ca7a55-74bd57dbd9-jrwrq\" (UID: \"a984df19-49d0-4c90-a51d-b1028220ee61\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ca7a55-74bd57dbd9-jrwrq"
Nov 01 05:26:08.637561 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:08.637539    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-ca7a55-74bd57dbd9-jrwrq"
Nov 01 05:26:08.766619 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:08.766587    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ca7a55-747b56569f-5s7gh"]
Nov 01 05:26:08.767552 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:26:08.767521    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod331a126d_11b4_4e43_88c3_5f2459793348.slice/crio-8a6242e9837d137f204fb0760391038bc3619dae9ab406fd5d861bdc26a819b3 WatchSource:0}: Error finding container 8a6242e9837d137f204fb0760391038bc3619dae9ab406fd5d861bdc26a819b3: Status 404 returned error can't find the container with id 8a6242e9837d137f204fb0760391038bc3619dae9ab406fd5d861bdc26a819b3
Nov 01 05:26:09.018885 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:09.018860    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ca7a55-74bd57dbd9-jrwrq"]
Nov 01 05:26:09.021001 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:26:09.020963    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda984df19_49d0_4c90_a51d_b1028220ee61.slice/crio-36e9eec07f1489d6fa6d61b955f3a83024970cbbe973064472483a5ac943b581 WatchSource:0}: Error finding container 36e9eec07f1489d6fa6d61b955f3a83024970cbbe973064472483a5ac943b581: Status 404 returned error can't find the container with id 36e9eec07f1489d6fa6d61b955f3a83024970cbbe973064472483a5ac943b581
Nov 01 05:26:09.156917 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:09.156886    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:26:09.747892 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:09.747855    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ca7a55-747b56569f-5s7gh" event={"ID":"331a126d-11b4-4e43-88c3-5f2459793348","Type":"ContainerStarted","Data":"16a12207a91113b992bb00d61786a7e70a0794347be0ca8af6b7f6dd7ae36a32"}
Nov 01 05:26:09.748103 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:09.747916    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ca7a55-747b56569f-5s7gh" event={"ID":"331a126d-11b4-4e43-88c3-5f2459793348","Type":"ContainerStarted","Data":"8a6242e9837d137f204fb0760391038bc3619dae9ab406fd5d861bdc26a819b3"}
Nov 01 05:26:09.748103 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:09.748018    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ca7a55-747b56569f-5s7gh"
Nov 01 05:26:09.749367 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:09.749342    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-ca7a55-74bd57dbd9-jrwrq" event={"ID":"a984df19-49d0-4c90-a51d-b1028220ee61","Type":"ContainerStarted","Data":"892e4c3e67a46b02f44c8fdaa1eb14f6e036ca66fa77559ab159deeaa52165ac"}
Nov 01 05:26:09.749520 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:09.749383    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-ca7a55-74bd57dbd9-jrwrq" event={"ID":"a984df19-49d0-4c90-a51d-b1028220ee61","Type":"ContainerStarted","Data":"36e9eec07f1489d6fa6d61b955f3a83024970cbbe973064472483a5ac943b581"}
Nov 01 05:26:09.749520 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:09.749416    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-ca7a55-74bd57dbd9-jrwrq"
Nov 01 05:26:09.865620 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:09.865573    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ca7a55-747b56569f-5s7gh" podStartSLOduration=1.865559285 podStartE2EDuration="1.865559285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:26:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:26:09.862113863 +0000 UTC m=+1347.932910584" watchObservedRunningTime="2026-11-01 05:26:09.865559285 +0000 UTC m=+1347.936356005"
Nov 01 05:26:09.920954 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:09.920930    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5lvz97"]
Nov 01 05:26:09.925025 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:09.925005    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5lvz97"
Nov 01 05:26:09.983192 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:09.982047    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2fd4c9\"/\"agc-cred-49dcf695f990\""
Nov 01 05:26:09.985715 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:09.985501    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-ca7a55-74bd57dbd9-jrwrq" podStartSLOduration=1.98548043 podStartE2EDuration="1.98548043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:26:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:26:09.984080277 +0000 UTC m=+1348.054877014" watchObservedRunningTime="2026-11-01 05:26:09.98548043 +0000 UTC m=+1348.056277153"
Nov 01 05:26:10.012886 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:10.012800    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5lvz97"]
Nov 01 05:26:10.033521 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:10.021855    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6775cee0-3876-46a8-b8f8-53b420c8277e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5lvz97\" (UID: \"6775cee0-3876-46a8-b8f8-53b420c8277e\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5lvz97"
Nov 01 05:26:10.033521 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:10.021911    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrhtj\" (UniqueName: \"kubernetes.io/projected/6775cee0-3876-46a8-b8f8-53b420c8277e-kube-api-access-lrhtj\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5lvz97\" (UID: \"6775cee0-3876-46a8-b8f8-53b420c8277e\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5lvz97"
Nov 01 05:26:10.033521 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:10.021959    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6775cee0-3876-46a8-b8f8-53b420c8277e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5lvz97\" (UID: \"6775cee0-3876-46a8-b8f8-53b420c8277e\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5lvz97"
Nov 01 05:26:10.033521 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:10.022031    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6775cee0-3876-46a8-b8f8-53b420c8277e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5lvz97\" (UID: \"6775cee0-3876-46a8-b8f8-53b420c8277e\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5lvz97"
Nov 01 05:26:10.141521 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:10.132417    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6775cee0-3876-46a8-b8f8-53b420c8277e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5lvz97\" (UID: \"6775cee0-3876-46a8-b8f8-53b420c8277e\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5lvz97"
Nov 01 05:26:10.141521 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:10.132492    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrhtj\" (UniqueName: \"kubernetes.io/projected/6775cee0-3876-46a8-b8f8-53b420c8277e-kube-api-access-lrhtj\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5lvz97\" (UID: \"6775cee0-3876-46a8-b8f8-53b420c8277e\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5lvz97"
Nov 01 05:26:10.141521 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:10.132550    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6775cee0-3876-46a8-b8f8-53b420c8277e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5lvz97\" (UID: \"6775cee0-3876-46a8-b8f8-53b420c8277e\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5lvz97"
Nov 01 05:26:10.141521 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:10.132608    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6775cee0-3876-46a8-b8f8-53b420c8277e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5lvz97\" (UID: \"6775cee0-3876-46a8-b8f8-53b420c8277e\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5lvz97"
Nov 01 05:26:10.141521 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:10.134068    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6775cee0-3876-46a8-b8f8-53b420c8277e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5lvz97\" (UID: \"6775cee0-3876-46a8-b8f8-53b420c8277e\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5lvz97"
Nov 01 05:26:10.141521 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:10.136258    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6775cee0-3876-46a8-b8f8-53b420c8277e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5lvz97\" (UID: \"6775cee0-3876-46a8-b8f8-53b420c8277e\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5lvz97"
Nov 01 05:26:10.141521 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:10.141160    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6775cee0-3876-46a8-b8f8-53b420c8277e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5lvz97\" (UID: \"6775cee0-3876-46a8-b8f8-53b420c8277e\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5lvz97"
Nov 01 05:26:10.196117 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:10.196090    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrhtj\" (UniqueName: \"kubernetes.io/projected/6775cee0-3876-46a8-b8f8-53b420c8277e-kube-api-access-lrhtj\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5lvz97\" (UID: \"6775cee0-3876-46a8-b8f8-53b420c8277e\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5lvz97"
Nov 01 05:26:10.239901 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:10.239876    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5lvz97"
Nov 01 05:26:10.407980 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:10.407938    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:26:10.620010 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:10.619985    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5lvz97"]
Nov 01 05:26:10.623328 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:26:10.623285    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6775cee0_3876_46a8_b8f8_53b420c8277e.slice/crio-c5f24bfb5da1707d150249fc6b9ca4f4478ad614cb038b2fe7b10e60612c2ec6 WatchSource:0}: Error finding container c5f24bfb5da1707d150249fc6b9ca4f4478ad614cb038b2fe7b10e60612c2ec6: Status 404 returned error can't find the container with id c5f24bfb5da1707d150249fc6b9ca4f4478ad614cb038b2fe7b10e60612c2ec6
Nov 01 05:26:10.756122 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:10.756081    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5lvz97" event={"ID":"6775cee0-3876-46a8-b8f8-53b420c8277e","Type":"ContainerStarted","Data":"47c1de4d5ca2711b84e2f129a93f2a2a66fc0a64d6e21e260878172d921d3739"}
Nov 01 05:26:10.756290 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:10.756140    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5lvz97" event={"ID":"6775cee0-3876-46a8-b8f8-53b420c8277e","Type":"ContainerStarted","Data":"c5f24bfb5da1707d150249fc6b9ca4f4478ad614cb038b2fe7b10e60612c2ec6"}
Nov 01 05:26:10.756445 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:10.756426    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5lvz97"
Nov 01 05:26:10.873769 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:10.873678    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5lvz97" podStartSLOduration=1.873665321 podStartE2EDuration="1.873665321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:26:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:26:10.872549179 +0000 UTC m=+1348.943345926" watchObservedRunningTime="2026-11-01 05:26:10.873665321 +0000 UTC m=+1348.944462041"
Nov 01 05:26:11.070685 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:11.070643    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:26:15.540559 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:15.540525    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:26:16.853896 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:16.853855    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:26:20.621829 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:20.621746    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:26:20.759773 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:20.759747    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ca7a55-747b56569f-5s7gh"
Nov 01 05:26:20.759954 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:20.759809    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-ca7a55-74bd57dbd9-jrwrq"
Nov 01 05:26:21.765147 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:21.765115    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5lvz97"
Nov 01 05:26:21.933203 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:21.932761    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6nz2mn"]
Nov 01 05:26:22.754748 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:22.754721    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:26:25.531284 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:25.531212    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:26:26.933594 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:26.933558    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6nz2mn" podUID="10e665b2-7fff-4260-9626-f7a834606eef" containerName="payload-processing" containerID="cri-o://862a0eedae263cf4efe90f4044a05fb34f90a2833e98d5d83dc7d8422b6331fc" gracePeriod=25
Nov 01 05:26:27.199919 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:27.199862    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6nz2mn"
Nov 01 05:26:27.265153 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:27.265125    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10e665b2-7fff-4260-9626-f7a834606eef-provider-credentials\" (UniqueName: \"kubernetes.io/projected/10e665b2-7fff-4260-9626-f7a834606eef-provider-credentials\") pod \"10e665b2-7fff-4260-9626-f7a834606eef\" (UID: \"10e665b2-7fff-4260-9626-f7a834606eef\") "
Nov 01 05:26:27.265318 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:27.265300    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10e665b2-7fff-4260-9626-f7a834606eef-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/10e665b2-7fff-4260-9626-f7a834606eef-routing-overlay-volume\") pod \"10e665b2-7fff-4260-9626-f7a834606eef\" (UID: \"10e665b2-7fff-4260-9626-f7a834606eef\") "
Nov 01 05:26:27.265438 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:27.265421    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10e665b2-7fff-4260-9626-f7a834606eef-kube-api-access-nqbpw\" (UniqueName: \"kubernetes.io/projected/10e665b2-7fff-4260-9626-f7a834606eef-kube-api-access-nqbpw\") pod \"10e665b2-7fff-4260-9626-f7a834606eef\" (UID: \"10e665b2-7fff-4260-9626-f7a834606eef\") "
Nov 01 05:26:27.265535 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:27.265501    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10e665b2-7fff-4260-9626-f7a834606eef-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10e665b2-7fff-4260-9626-f7a834606eef-plugins-config-volume\") pod \"10e665b2-7fff-4260-9626-f7a834606eef\" (UID: \"10e665b2-7fff-4260-9626-f7a834606eef\") "
Nov 01 05:26:27.265689 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:27.265662    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10e665b2-7fff-4260-9626-f7a834606eef-routing-overlay-volume" pod "10e665b2-7fff-4260-9626-f7a834606eef" (UID: "10e665b2-7fff-4260-9626-f7a834606eef"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:26:27.265911 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:27.265888    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10e665b2-7fff-4260-9626-f7a834606eef-plugins-config-volume" pod "10e665b2-7fff-4260-9626-f7a834606eef" (UID: "10e665b2-7fff-4260-9626-f7a834606eef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:26:27.266010 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:27.265993    5689 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/10e665b2-7fff-4260-9626-f7a834606eef-routing-overlay-volume\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:26:27.266010 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:27.266007    5689 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10e665b2-7fff-4260-9626-f7a834606eef-plugins-config-volume\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:26:27.267570 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:27.267550    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10e665b2-7fff-4260-9626-f7a834606eef-provider-credentials" pod "10e665b2-7fff-4260-9626-f7a834606eef" (UID: "10e665b2-7fff-4260-9626-f7a834606eef"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:26:27.267643 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:27.267595    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10e665b2-7fff-4260-9626-f7a834606eef-kube-api-access-nqbpw" pod "10e665b2-7fff-4260-9626-f7a834606eef" (UID: "10e665b2-7fff-4260-9626-f7a834606eef"). InnerVolumeSpecName "kube-api-access-nqbpw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:26:27.367024 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:27.366993    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqbpw\" (UniqueName: \"kubernetes.io/projected/10e665b2-7fff-4260-9626-f7a834606eef-kube-api-access-nqbpw\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:26:27.367024 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:27.367018    5689 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/10e665b2-7fff-4260-9626-f7a834606eef-provider-credentials\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:26:27.855918 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:27.855885    5689 generic.go:356] "Generic (PLEG): container finished" podID="10e665b2-7fff-4260-9626-f7a834606eef" containerID="862a0eedae263cf4efe90f4044a05fb34f90a2833e98d5d83dc7d8422b6331fc" exitCode=0
Nov 01 05:26:27.856065 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:27.855930    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6nz2mn"
Nov 01 05:26:27.856065 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:27.855962    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6nz2mn" event={"ID":"10e665b2-7fff-4260-9626-f7a834606eef","Type":"ContainerDied","Data":"862a0eedae263cf4efe90f4044a05fb34f90a2833e98d5d83dc7d8422b6331fc"}
Nov 01 05:26:27.856065 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:27.856019    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6nz2mn" event={"ID":"10e665b2-7fff-4260-9626-f7a834606eef","Type":"ContainerDied","Data":"92d95367f45cdbe50f01d67c7462666082e9a64c1495626fe02f7f4cf21539fb"}
Nov 01 05:26:27.856065 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:27.856048    5689 scope.go:122] "RemoveContainer" containerID="862a0eedae263cf4efe90f4044a05fb34f90a2833e98d5d83dc7d8422b6331fc"
Nov 01 05:26:27.866279 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:27.866265    5689 scope.go:122] "RemoveContainer" containerID="862a0eedae263cf4efe90f4044a05fb34f90a2833e98d5d83dc7d8422b6331fc"
Nov 01 05:26:27.866542 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:26:27.866524    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"862a0eedae263cf4efe90f4044a05fb34f90a2833e98d5d83dc7d8422b6331fc\": container with ID starting with 862a0eedae263cf4efe90f4044a05fb34f90a2833e98d5d83dc7d8422b6331fc not found: ID does not exist" containerID="862a0eedae263cf4efe90f4044a05fb34f90a2833e98d5d83dc7d8422b6331fc"
Nov 01 05:26:27.866600 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:27.866550    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"862a0eedae263cf4efe90f4044a05fb34f90a2833e98d5d83dc7d8422b6331fc"} err="failed to get container status \"862a0eedae263cf4efe90f4044a05fb34f90a2833e98d5d83dc7d8422b6331fc\": rpc error: code = NotFound desc = could not find container \"862a0eedae263cf4efe90f4044a05fb34f90a2833e98d5d83dc7d8422b6331fc\": container with ID starting with 862a0eedae263cf4efe90f4044a05fb34f90a2833e98d5d83dc7d8422b6331fc not found: ID does not exist"
Nov 01 05:26:28.019627 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:28.019594    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6nz2mn"]
Nov 01 05:26:28.021875 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:28.021854    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6nz2mn"]
Nov 01 05:26:28.067504 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:28.067448    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10e665b2-7fff-4260-9626-f7a834606eef" path="/var/lib/kubelet/pods/10e665b2-7fff-4260-9626-f7a834606eef/volumes"
Nov 01 05:26:30.074816 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:26:30.074775    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:26:31.424297 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:31.424263    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:26:34.359443 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:34.359411    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:26:36.694051 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:36.694023    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:26:41.439303 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:41.439238    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:26:43.430229 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:43.430198    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:26:44.806186 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:44.806154    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:26:45.483489 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:45.465877    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:26:45.737037 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:45.736935    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:26:46.493370 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:46.493342    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:26:46.674904 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:46.674861    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:26:46.853131 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:46.853047    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:26:48.768969 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:48.768936    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:26:48.962550 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:48.962518    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:26:49.413559 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:49.413526    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:26:49.594635 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:49.594606    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:26:49.785038 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:49.784920    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:26:52.042418 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:52.042381    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:26:52.635714 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:52.635680    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:26:52.821930 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:52.821903    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:26:53.195931 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:53.195889    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-c6e99c-8b7bf895f-hj225"]
Nov 01 05:26:53.197405 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:53.197374    5689 cpu_manager.go:414] "containerMap: removing container" podUID="10e665b2-7fff-4260-9626-f7a834606eef" containerName="payload-processing"
Nov 01 05:26:53.197604 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:53.197586    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10e665b2-7fff-4260-9626-f7a834606eef" containerName="payload-processing"
Nov 01 05:26:53.197902 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:53.197887    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="10e665b2-7fff-4260-9626-f7a834606eef" containerName="payload-processing"
Nov 01 05:26:53.224289 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:53.224258    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-c6e99c-8b7bf895f-hj225"
Nov 01 05:26:53.257054 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:53.257018    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4j27\" (UniqueName: \"kubernetes.io/projected/7a44bf0d-d385-4d71-9b9a-b0c8d86f4c48-kube-api-access-r4j27\") pod \"payload-processing-e2e-extmod-b-c6e99c-8b7bf895f-hj225\" (UID: \"7a44bf0d-d385-4d71-9b9a-b0c8d86f4c48\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-c6e99c-8b7bf895f-hj225"
Nov 01 05:26:53.257541 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:53.257504    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a44bf0d-d385-4d71-9b9a-b0c8d86f4c48-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-c6e99c-8b7bf895f-hj225\" (UID: \"7a44bf0d-d385-4d71-9b9a-b0c8d86f4c48\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-c6e99c-8b7bf895f-hj225"
Nov 01 05:26:53.258196 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:53.258178    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-c6e99c-8b7bf895f-hj225"]
Nov 01 05:26:53.300798 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:53.300759    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-c6e99c-dockercfg-7tzjt\""
Nov 01 05:26:53.301110 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:53.301092    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-c6e99c\""
Nov 01 05:26:53.349664 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:53.349631    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-c6e99c-fcb666fb7-kc9cw"]
Nov 01 05:26:53.353627 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:53.353609    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c6e99c-fcb666fb7-kc9cw"
Nov 01 05:26:53.358676 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:53.358649    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4j27\" (UniqueName: \"kubernetes.io/projected/7a44bf0d-d385-4d71-9b9a-b0c8d86f4c48-kube-api-access-r4j27\") pod \"payload-processing-e2e-extmod-b-c6e99c-8b7bf895f-hj225\" (UID: \"7a44bf0d-d385-4d71-9b9a-b0c8d86f4c48\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-c6e99c-8b7bf895f-hj225"
Nov 01 05:26:53.358815 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:53.358776    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a44bf0d-d385-4d71-9b9a-b0c8d86f4c48-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-c6e99c-8b7bf895f-hj225\" (UID: \"7a44bf0d-d385-4d71-9b9a-b0c8d86f4c48\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-c6e99c-8b7bf895f-hj225"
Nov 01 05:26:53.359675 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:53.359653    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a44bf0d-d385-4d71-9b9a-b0c8d86f4c48-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-c6e99c-8b7bf895f-hj225\" (UID: \"7a44bf0d-d385-4d71-9b9a-b0c8d86f4c48\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-c6e99c-8b7bf895f-hj225"
Nov 01 05:26:53.412170 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:53.412139    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-c6e99c-fcb666fb7-kc9cw"]
Nov 01 05:26:53.420101 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:53.420066    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4j27\" (UniqueName: \"kubernetes.io/projected/7a44bf0d-d385-4d71-9b9a-b0c8d86f4c48-kube-api-access-r4j27\") pod \"payload-processing-e2e-extmod-b-c6e99c-8b7bf895f-hj225\" (UID: \"7a44bf0d-d385-4d71-9b9a-b0c8d86f4c48\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-c6e99c-8b7bf895f-hj225"
Nov 01 05:26:53.459602 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:53.459540    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmf76\" (UniqueName: \"kubernetes.io/projected/8420da2d-22af-4a07-9054-220111bd91bf-kube-api-access-vmf76\") pod \"payload-pre-processing-e2e-extmod-b-c6e99c-fcb666fb7-kc9cw\" (UID: \"8420da2d-22af-4a07-9054-220111bd91bf\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c6e99c-fcb666fb7-kc9cw"
Nov 01 05:26:53.459716 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:53.459602    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8420da2d-22af-4a07-9054-220111bd91bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-c6e99c-fcb666fb7-kc9cw\" (UID: \"8420da2d-22af-4a07-9054-220111bd91bf\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c6e99c-fcb666fb7-kc9cw"
Nov 01 05:26:53.561192 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:53.561159    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmf76\" (UniqueName: \"kubernetes.io/projected/8420da2d-22af-4a07-9054-220111bd91bf-kube-api-access-vmf76\") pod \"payload-pre-processing-e2e-extmod-b-c6e99c-fcb666fb7-kc9cw\" (UID: \"8420da2d-22af-4a07-9054-220111bd91bf\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c6e99c-fcb666fb7-kc9cw"
Nov 01 05:26:53.561340 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:53.561233    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8420da2d-22af-4a07-9054-220111bd91bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-c6e99c-fcb666fb7-kc9cw\" (UID: \"8420da2d-22af-4a07-9054-220111bd91bf\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c6e99c-fcb666fb7-kc9cw"
Nov 01 05:26:53.562019 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:53.561998    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8420da2d-22af-4a07-9054-220111bd91bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-c6e99c-fcb666fb7-kc9cw\" (UID: \"8420da2d-22af-4a07-9054-220111bd91bf\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c6e99c-fcb666fb7-kc9cw"
Nov 01 05:26:53.617921 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:53.617892    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-c6e99c-8b7bf895f-hj225"
Nov 01 05:26:53.622013 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:53.621988    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmf76\" (UniqueName: \"kubernetes.io/projected/8420da2d-22af-4a07-9054-220111bd91bf-kube-api-access-vmf76\") pod \"payload-pre-processing-e2e-extmod-b-c6e99c-fcb666fb7-kc9cw\" (UID: \"8420da2d-22af-4a07-9054-220111bd91bf\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c6e99c-fcb666fb7-kc9cw"
Nov 01 05:26:53.668303 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:53.668276    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c6e99c-fcb666fb7-kc9cw"
Nov 01 05:26:53.997874 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:53.997852    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-c6e99c-8b7bf895f-hj225"]
Nov 01 05:26:54.000410 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:26:54.000381    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a44bf0d_d385_4d71_9b9a_b0c8d86f4c48.slice/crio-26da8973aea4dea3f4ec919ddc09a1056f26038941f945836e33589b4e7f9590 WatchSource:0}: Error finding container 26da8973aea4dea3f4ec919ddc09a1056f26038941f945836e33589b4e7f9590: Status 404 returned error can't find the container with id 26da8973aea4dea3f4ec919ddc09a1056f26038941f945836e33589b4e7f9590
Nov 01 05:26:54.020261 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:54.020231    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-c6e99c-8b7bf895f-hj225" event={"ID":"7a44bf0d-d385-4d71-9b9a-b0c8d86f4c48","Type":"ContainerStarted","Data":"26da8973aea4dea3f4ec919ddc09a1056f26038941f945836e33589b4e7f9590"}
Nov 01 05:26:54.111117 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:54.111095    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-c6e99c-fcb666fb7-kc9cw"]
Nov 01 05:26:54.113631 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:26:54.113608    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8420da2d_22af_4a07_9054_220111bd91bf.slice/crio-c243794209dd0d46a520e557fd3aa951e0f752745fc495795f098b813677bd69 WatchSource:0}: Error finding container c243794209dd0d46a520e557fd3aa951e0f752745fc495795f098b813677bd69: Status 404 returned error can't find the container with id c243794209dd0d46a520e557fd3aa951e0f752745fc495795f098b813677bd69
Nov 01 05:26:55.027375 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:55.027334    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-c6e99c-8b7bf895f-hj225" event={"ID":"7a44bf0d-d385-4d71-9b9a-b0c8d86f4c48","Type":"ContainerStarted","Data":"0338c714995b2a99b57d637d8d23baaa0168a76cdd0d9e59e9e3e7bac49819bb"}
Nov 01 05:26:55.027844 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:55.027414    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-c6e99c-8b7bf895f-hj225"
Nov 01 05:26:55.029004 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:55.028981    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c6e99c-fcb666fb7-kc9cw" event={"ID":"8420da2d-22af-4a07-9054-220111bd91bf","Type":"ContainerStarted","Data":"605ced9811813726efc1f6ee1752e066e1965dd7daed5d9160fceeb7ecae5bec"}
Nov 01 05:26:55.029066 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:55.029029    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c6e99c-fcb666fb7-kc9cw" event={"ID":"8420da2d-22af-4a07-9054-220111bd91bf","Type":"ContainerStarted","Data":"c243794209dd0d46a520e557fd3aa951e0f752745fc495795f098b813677bd69"}
Nov 01 05:26:55.029066 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:55.029052    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c6e99c-fcb666fb7-kc9cw"
Nov 01 05:26:55.145147 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:55.145098    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-c6e99c-8b7bf895f-hj225" podStartSLOduration=2.145086872 podStartE2EDuration="2.145086872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:26:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:26:55.142643373 +0000 UTC m=+1393.213440095" watchObservedRunningTime="2026-11-01 05:26:55.145086872 +0000 UTC m=+1393.215883591"
Nov 01 05:26:55.263051 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:55.262979    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c6e99c-fcb666fb7-kc9cw" podStartSLOduration=2.262962091 podStartE2EDuration="2.262962091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:26:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:26:55.258354209 +0000 UTC m=+1393.329150971" watchObservedRunningTime="2026-11-01 05:26:55.262962091 +0000 UTC m=+1393.333758812"
Nov 01 05:26:56.904539 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:56.904505    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:26:57.753102 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:26:57.753062    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:27:00.069910 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:27:00.069871    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:27:01.525653 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:01.525602    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:27:02.023370 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:02.023337    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:27:06.038240 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:06.038210    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c6e99c-fcb666fb7-kc9cw"
Nov 01 05:27:06.038686 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:06.038279    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-c6e99c-8b7bf895f-hj225"
Nov 01 05:27:06.926070 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:06.926037    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:27:11.714387 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:11.714355    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:27:15.750985 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:15.750955    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2864b6-6d657dbf4d-n6lvj"]
Nov 01 05:27:15.751391 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:15.751167    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2864b6-6d657dbf4d-n6lvj" podUID="137ed9b9-0685-4ac6-a78e-e76348d77583" containerName="maas-api" containerID="cri-o://4e3f021fc9263beb34dc693d444e61487df36d1d0497752f23fba396ef368eab" gracePeriod=30
Nov 01 05:27:15.791911 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:15.791877    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:27:15.976085 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:15.975757    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:27:16.086639 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:16.086606    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2864b6-6d657dbf4d-n6lvj"
Nov 01 05:27:16.162386 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:16.162361    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/137ed9b9-0685-4ac6-a78e-e76348d77583-metrics-certs\" (UniqueName: \"kubernetes.io/secret/137ed9b9-0685-4ac6-a78e-e76348d77583-metrics-certs\") pod \"137ed9b9-0685-4ac6-a78e-e76348d77583\" (UID: \"137ed9b9-0685-4ac6-a78e-e76348d77583\") "
Nov 01 05:27:16.162579 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:16.162399    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/137ed9b9-0685-4ac6-a78e-e76348d77583-kube-api-access-fr92r\" (UniqueName: \"kubernetes.io/projected/137ed9b9-0685-4ac6-a78e-e76348d77583-kube-api-access-fr92r\") pod \"137ed9b9-0685-4ac6-a78e-e76348d77583\" (UID: \"137ed9b9-0685-4ac6-a78e-e76348d77583\") "
Nov 01 05:27:16.162579 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:16.162501    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/137ed9b9-0685-4ac6-a78e-e76348d77583-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/137ed9b9-0685-4ac6-a78e-e76348d77583-maas-api-tls\") pod \"137ed9b9-0685-4ac6-a78e-e76348d77583\" (UID: \"137ed9b9-0685-4ac6-a78e-e76348d77583\") "
Nov 01 05:27:16.162767 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:16.162751    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/137ed9b9-0685-4ac6-a78e-e76348d77583-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/137ed9b9-0685-4ac6-a78e-e76348d77583-openshift-service-ca\") pod \"137ed9b9-0685-4ac6-a78e-e76348d77583\" (UID: \"137ed9b9-0685-4ac6-a78e-e76348d77583\") "
Nov 01 05:27:16.163092 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:16.163066    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/137ed9b9-0685-4ac6-a78e-e76348d77583-openshift-service-ca" pod "137ed9b9-0685-4ac6-a78e-e76348d77583" (UID: "137ed9b9-0685-4ac6-a78e-e76348d77583"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:27:16.163287 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:16.163270    5689 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/137ed9b9-0685-4ac6-a78e-e76348d77583-openshift-service-ca\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:27:16.164630 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:16.164606    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/137ed9b9-0685-4ac6-a78e-e76348d77583-kube-api-access-fr92r" pod "137ed9b9-0685-4ac6-a78e-e76348d77583" (UID: "137ed9b9-0685-4ac6-a78e-e76348d77583"). InnerVolumeSpecName "kube-api-access-fr92r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:27:16.165116 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:16.165091    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/137ed9b9-0685-4ac6-a78e-e76348d77583-maas-api-tls" pod "137ed9b9-0685-4ac6-a78e-e76348d77583" (UID: "137ed9b9-0685-4ac6-a78e-e76348d77583"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 05:27:16.165208 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:16.165112    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/137ed9b9-0685-4ac6-a78e-e76348d77583-metrics-certs" pod "137ed9b9-0685-4ac6-a78e-e76348d77583" (UID: "137ed9b9-0685-4ac6-a78e-e76348d77583"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 05:27:16.184267 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:16.184244    5689 generic.go:356] "Generic (PLEG): container finished" podID="137ed9b9-0685-4ac6-a78e-e76348d77583" containerID="4e3f021fc9263beb34dc693d444e61487df36d1d0497752f23fba396ef368eab" exitCode=0
Nov 01 05:27:16.184367 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:16.184323    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2864b6-6d657dbf4d-n6lvj" event={"ID":"137ed9b9-0685-4ac6-a78e-e76348d77583","Type":"ContainerDied","Data":"4e3f021fc9263beb34dc693d444e61487df36d1d0497752f23fba396ef368eab"}
Nov 01 05:27:16.184367 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:16.184336    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2864b6-6d657dbf4d-n6lvj"
Nov 01 05:27:16.184367 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:16.184359    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2864b6-6d657dbf4d-n6lvj" event={"ID":"137ed9b9-0685-4ac6-a78e-e76348d77583","Type":"ContainerDied","Data":"25f6214f5257fae514c3f248c067cddb2a1cd0223bc1679785635c5f6a7c58bd"}
Nov 01 05:27:16.184508 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:16.184380    5689 scope.go:122] "RemoveContainer" containerID="4e3f021fc9263beb34dc693d444e61487df36d1d0497752f23fba396ef368eab"
Nov 01 05:27:16.194046 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:16.194031    5689 scope.go:122] "RemoveContainer" containerID="4e3f021fc9263beb34dc693d444e61487df36d1d0497752f23fba396ef368eab"
Nov 01 05:27:16.194298 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:27:16.194281    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e3f021fc9263beb34dc693d444e61487df36d1d0497752f23fba396ef368eab\": container with ID starting with 4e3f021fc9263beb34dc693d444e61487df36d1d0497752f23fba396ef368eab not found: ID does not exist" containerID="4e3f021fc9263beb34dc693d444e61487df36d1d0497752f23fba396ef368eab"
Nov 01 05:27:16.194344 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:16.194306    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e3f021fc9263beb34dc693d444e61487df36d1d0497752f23fba396ef368eab"} err="failed to get container status \"4e3f021fc9263beb34dc693d444e61487df36d1d0497752f23fba396ef368eab\": rpc error: code = NotFound desc = could not find container \"4e3f021fc9263beb34dc693d444e61487df36d1d0497752f23fba396ef368eab\": container with ID starting with 4e3f021fc9263beb34dc693d444e61487df36d1d0497752f23fba396ef368eab not found: ID does not exist"
Nov 01 05:27:16.219523 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:16.219498    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2864b6-6d848878b6-zcftx"]
Nov 01 05:27:16.219726 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:16.219706    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-2864b6-6d848878b6-zcftx" podUID="390739de-f035-41a9-b2d9-ff8664766ba6" containerName="payload-processing" containerID="cri-o://f780cbd612d523e2f357ecb82488390a61f2a852c80746ad8f68c6244a61a50e" gracePeriod=30
Nov 01 05:27:16.264243 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:16.264221    5689 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/137ed9b9-0685-4ac6-a78e-e76348d77583-metrics-certs\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:27:16.264243 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:16.264244    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fr92r\" (UniqueName: \"kubernetes.io/projected/137ed9b9-0685-4ac6-a78e-e76348d77583-kube-api-access-fr92r\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:27:16.264399 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:16.264255    5689 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/137ed9b9-0685-4ac6-a78e-e76348d77583-maas-api-tls\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:27:16.269626 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:16.269600    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2864b6-669cd47975-bfwj8"]
Nov 01 05:27:16.269843 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:16.269820    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2864b6-669cd47975-bfwj8" podUID="3642df08-0365-4562-b5f0-e480f50c2761" containerName="payload-pre-processing" containerID="cri-o://6d3fddc8e4e1fdd317a6b08d93b90f7fb0182298f8723ef9446efae524c6b4d5" gracePeriod=30
Nov 01 05:27:16.320518 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:16.319863    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2864b6-6d657dbf4d-n6lvj"]
Nov 01 05:27:16.323447 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:16.323422    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2864b6-6d657dbf4d-n6lvj"]
Nov 01 05:27:16.412411 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:16.403763    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:27:16.534341 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:16.534311    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2864b6-6d848878b6-zcftx"
Nov 01 05:27:16.567727 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:16.567706    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2864b6-669cd47975-bfwj8"
Nov 01 05:27:16.568170 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:16.568155    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/390739de-f035-41a9-b2d9-ff8664766ba6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/390739de-f035-41a9-b2d9-ff8664766ba6-plugins-config-volume\") pod \"390739de-f035-41a9-b2d9-ff8664766ba6\" (UID: \"390739de-f035-41a9-b2d9-ff8664766ba6\") "
Nov 01 05:27:16.568282 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:16.568267    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/390739de-f035-41a9-b2d9-ff8664766ba6-kube-api-access-rh48d\" (UniqueName: \"kubernetes.io/projected/390739de-f035-41a9-b2d9-ff8664766ba6-kube-api-access-rh48d\") pod \"390739de-f035-41a9-b2d9-ff8664766ba6\" (UID: \"390739de-f035-41a9-b2d9-ff8664766ba6\") "
Nov 01 05:27:16.568538 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:16.568516    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/390739de-f035-41a9-b2d9-ff8664766ba6-plugins-config-volume" pod "390739de-f035-41a9-b2d9-ff8664766ba6" (UID: "390739de-f035-41a9-b2d9-ff8664766ba6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:27:16.568817 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:16.568799    5689 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/390739de-f035-41a9-b2d9-ff8664766ba6-plugins-config-volume\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:27:16.570322 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:16.570305    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/390739de-f035-41a9-b2d9-ff8664766ba6-kube-api-access-rh48d" pod "390739de-f035-41a9-b2d9-ff8664766ba6" (UID: "390739de-f035-41a9-b2d9-ff8664766ba6"). InnerVolumeSpecName "kube-api-access-rh48d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:27:16.582890 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:16.582839    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:27:16.669362 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:16.669331    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3642df08-0365-4562-b5f0-e480f50c2761-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3642df08-0365-4562-b5f0-e480f50c2761-plugins-config-volume\") pod \"3642df08-0365-4562-b5f0-e480f50c2761\" (UID: \"3642df08-0365-4562-b5f0-e480f50c2761\") "
Nov 01 05:27:16.669530 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:16.669515    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3642df08-0365-4562-b5f0-e480f50c2761-kube-api-access-rjw7q\" (UniqueName: \"kubernetes.io/projected/3642df08-0365-4562-b5f0-e480f50c2761-kube-api-access-rjw7q\") pod \"3642df08-0365-4562-b5f0-e480f50c2761\" (UID: \"3642df08-0365-4562-b5f0-e480f50c2761\") "
Nov 01 05:27:16.669755 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:16.669729    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3642df08-0365-4562-b5f0-e480f50c2761-plugins-config-volume" pod "3642df08-0365-4562-b5f0-e480f50c2761" (UID: "3642df08-0365-4562-b5f0-e480f50c2761"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:27:16.669961 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:16.669936    5689 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3642df08-0365-4562-b5f0-e480f50c2761-plugins-config-volume\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:27:16.670109 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:16.669969    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rh48d\" (UniqueName: \"kubernetes.io/projected/390739de-f035-41a9-b2d9-ff8664766ba6-kube-api-access-rh48d\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:27:16.671750 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:16.671727    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3642df08-0365-4562-b5f0-e480f50c2761-kube-api-access-rjw7q" pod "3642df08-0365-4562-b5f0-e480f50c2761" (UID: "3642df08-0365-4562-b5f0-e480f50c2761"). InnerVolumeSpecName "kube-api-access-rjw7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:27:16.776847 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:16.772788    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjw7q\" (UniqueName: \"kubernetes.io/projected/3642df08-0365-4562-b5f0-e480f50c2761-kube-api-access-rjw7q\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:27:17.191439 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:17.191408    5689 generic.go:356] "Generic (PLEG): container finished" podID="390739de-f035-41a9-b2d9-ff8664766ba6" containerID="f780cbd612d523e2f357ecb82488390a61f2a852c80746ad8f68c6244a61a50e" exitCode=0
Nov 01 05:27:17.191654 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:17.191479    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2864b6-6d848878b6-zcftx"
Nov 01 05:27:17.191654 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:17.191492    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2864b6-6d848878b6-zcftx" event={"ID":"390739de-f035-41a9-b2d9-ff8664766ba6","Type":"ContainerDied","Data":"f780cbd612d523e2f357ecb82488390a61f2a852c80746ad8f68c6244a61a50e"}
Nov 01 05:27:17.191654 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:17.191539    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2864b6-6d848878b6-zcftx" event={"ID":"390739de-f035-41a9-b2d9-ff8664766ba6","Type":"ContainerDied","Data":"2e21a8a185541d60d8d0f2c3d5c686e2d4d1db1750de631bf68049029e84b558"}
Nov 01 05:27:17.191654 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:17.191560    5689 scope.go:122] "RemoveContainer" containerID="f780cbd612d523e2f357ecb82488390a61f2a852c80746ad8f68c6244a61a50e"
Nov 01 05:27:17.193005 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:17.192984    5689 generic.go:356] "Generic (PLEG): container finished" podID="3642df08-0365-4562-b5f0-e480f50c2761" containerID="6d3fddc8e4e1fdd317a6b08d93b90f7fb0182298f8723ef9446efae524c6b4d5" exitCode=0
Nov 01 05:27:17.193100 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:17.193021    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2864b6-669cd47975-bfwj8"
Nov 01 05:27:17.193100 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:17.193060    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2864b6-669cd47975-bfwj8" event={"ID":"3642df08-0365-4562-b5f0-e480f50c2761","Type":"ContainerDied","Data":"6d3fddc8e4e1fdd317a6b08d93b90f7fb0182298f8723ef9446efae524c6b4d5"}
Nov 01 05:27:17.193177 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:17.193120    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2864b6-669cd47975-bfwj8" event={"ID":"3642df08-0365-4562-b5f0-e480f50c2761","Type":"ContainerDied","Data":"6f56eb3e0a459047e25eb7eed1d3d729ff8f20e9ef5c60598405a3cadf2b47d5"}
Nov 01 05:27:17.202390 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:17.202356    5689 scope.go:122] "RemoveContainer" containerID="f780cbd612d523e2f357ecb82488390a61f2a852c80746ad8f68c6244a61a50e"
Nov 01 05:27:17.202684 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:27:17.202661    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f780cbd612d523e2f357ecb82488390a61f2a852c80746ad8f68c6244a61a50e\": container with ID starting with f780cbd612d523e2f357ecb82488390a61f2a852c80746ad8f68c6244a61a50e not found: ID does not exist" containerID="f780cbd612d523e2f357ecb82488390a61f2a852c80746ad8f68c6244a61a50e"
Nov 01 05:27:17.202786 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:17.202694    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f780cbd612d523e2f357ecb82488390a61f2a852c80746ad8f68c6244a61a50e"} err="failed to get container status \"f780cbd612d523e2f357ecb82488390a61f2a852c80746ad8f68c6244a61a50e\": rpc error: code = NotFound desc = could not find container \"f780cbd612d523e2f357ecb82488390a61f2a852c80746ad8f68c6244a61a50e\": container with ID starting with f780cbd612d523e2f357ecb82488390a61f2a852c80746ad8f68c6244a61a50e not found: ID does not exist"
Nov 01 05:27:17.202786 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:17.202714    5689 scope.go:122] "RemoveContainer" containerID="6d3fddc8e4e1fdd317a6b08d93b90f7fb0182298f8723ef9446efae524c6b4d5"
Nov 01 05:27:17.211791 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:17.211772    5689 scope.go:122] "RemoveContainer" containerID="6d3fddc8e4e1fdd317a6b08d93b90f7fb0182298f8723ef9446efae524c6b4d5"
Nov 01 05:27:17.212016 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:27:17.211998    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d3fddc8e4e1fdd317a6b08d93b90f7fb0182298f8723ef9446efae524c6b4d5\": container with ID starting with 6d3fddc8e4e1fdd317a6b08d93b90f7fb0182298f8723ef9446efae524c6b4d5 not found: ID does not exist" containerID="6d3fddc8e4e1fdd317a6b08d93b90f7fb0182298f8723ef9446efae524c6b4d5"
Nov 01 05:27:17.212080 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:17.212025    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d3fddc8e4e1fdd317a6b08d93b90f7fb0182298f8723ef9446efae524c6b4d5"} err="failed to get container status \"6d3fddc8e4e1fdd317a6b08d93b90f7fb0182298f8723ef9446efae524c6b4d5\": rpc error: code = NotFound desc = could not find container \"6d3fddc8e4e1fdd317a6b08d93b90f7fb0182298f8723ef9446efae524c6b4d5\": container with ID starting with 6d3fddc8e4e1fdd317a6b08d93b90f7fb0182298f8723ef9446efae524c6b4d5 not found: ID does not exist"
Nov 01 05:27:17.355723 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:17.355689    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2864b6-669cd47975-bfwj8"]
Nov 01 05:27:17.359202 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:17.359177    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2864b6-669cd47975-bfwj8"]
Nov 01 05:27:17.469374 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:17.469249    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2864b6-6d848878b6-zcftx"]
Nov 01 05:27:17.471935 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:17.471896    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2864b6-6d848878b6-zcftx"]
Nov 01 05:27:17.625242 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:17.625214    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:27:17.858697 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:17.850198    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:27:18.068444 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:18.068412    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="137ed9b9-0685-4ac6-a78e-e76348d77583" path="/var/lib/kubelet/pods/137ed9b9-0685-4ac6-a78e-e76348d77583/volumes"
Nov 01 05:27:18.068876 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:18.068860    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3642df08-0365-4562-b5f0-e480f50c2761" path="/var/lib/kubelet/pods/3642df08-0365-4562-b5f0-e480f50c2761/volumes"
Nov 01 05:27:18.069199 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:18.069187    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="390739de-f035-41a9-b2d9-ff8664766ba6" path="/var/lib/kubelet/pods/390739de-f035-41a9-b2d9-ff8664766ba6/volumes"
Nov 01 05:27:18.108654 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:18.108624    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:27:21.652674 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:21.652635    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:27:22.311279 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:22.311251    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:27:22.487700 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:22.487673    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:27:23.924828 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:23.924796    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:27:24.516658 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:24.516625    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:27:29.298986 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:29.298950    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:27:29.311812 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:29.311785    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:27:29.765323 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:29.765293    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:27:30.069414 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:27:30.069338    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:27:34.412976 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:34.412942    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:27:39.200654 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:39.200620    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:27:43.634575 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:43.634540    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-c6e99c-bbf76fcb8-bbrnt"]
Nov 01 05:27:43.635123 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:43.635109    5689 cpu_manager.go:414] "containerMap: removing container" podUID="137ed9b9-0685-4ac6-a78e-e76348d77583" containerName="maas-api"
Nov 01 05:27:43.635184 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:43.635124    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="137ed9b9-0685-4ac6-a78e-e76348d77583" containerName="maas-api"
Nov 01 05:27:43.635184 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:43.635137    5689 cpu_manager.go:414] "containerMap: removing container" podUID="390739de-f035-41a9-b2d9-ff8664766ba6" containerName="payload-processing"
Nov 01 05:27:43.635184 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:43.635143    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="390739de-f035-41a9-b2d9-ff8664766ba6" containerName="payload-processing"
Nov 01 05:27:43.635184 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:43.635155    5689 cpu_manager.go:414] "containerMap: removing container" podUID="3642df08-0365-4562-b5f0-e480f50c2761" containerName="payload-pre-processing"
Nov 01 05:27:43.635184 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:43.635161    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3642df08-0365-4562-b5f0-e480f50c2761" containerName="payload-pre-processing"
Nov 01 05:27:43.635337 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:43.635241    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="137ed9b9-0685-4ac6-a78e-e76348d77583" containerName="maas-api"
Nov 01 05:27:43.635337 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:43.635248    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="390739de-f035-41a9-b2d9-ff8664766ba6" containerName="payload-processing"
Nov 01 05:27:43.635337 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:43.635256    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="3642df08-0365-4562-b5f0-e480f50c2761" containerName="payload-pre-processing"
Nov 01 05:27:43.639656 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:43.639640    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c6e99c-bbf76fcb8-bbrnt"
Nov 01 05:27:43.691619 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:43.691595    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-c6e99c-serving-cert\""
Nov 01 05:27:43.701493 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:43.699389    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9729d894-945b-4d8b-8700-46d3552c2c8e-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-c6e99c-bbf76fcb8-bbrnt\" (UID: \"9729d894-945b-4d8b-8700-46d3552c2c8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c6e99c-bbf76fcb8-bbrnt"
Nov 01 05:27:43.701493 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:43.699523    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9729d894-945b-4d8b-8700-46d3552c2c8e-metrics-certs\") pod \"maas-api-e2e-extmod-b-c6e99c-bbf76fcb8-bbrnt\" (UID: \"9729d894-945b-4d8b-8700-46d3552c2c8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c6e99c-bbf76fcb8-bbrnt"
Nov 01 05:27:43.701493 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:43.699600    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fwp7\" (UniqueName: \"kubernetes.io/projected/9729d894-945b-4d8b-8700-46d3552c2c8e-kube-api-access-6fwp7\") pod \"maas-api-e2e-extmod-b-c6e99c-bbf76fcb8-bbrnt\" (UID: \"9729d894-945b-4d8b-8700-46d3552c2c8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c6e99c-bbf76fcb8-bbrnt"
Nov 01 05:27:43.701493 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:43.699674    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9729d894-945b-4d8b-8700-46d3552c2c8e-maas-api-tls\") pod \"maas-api-e2e-extmod-b-c6e99c-bbf76fcb8-bbrnt\" (UID: \"9729d894-945b-4d8b-8700-46d3552c2c8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c6e99c-bbf76fcb8-bbrnt"
Nov 01 05:27:43.743763 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:43.743735    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-c6e99c-bbf76fcb8-bbrnt"]
Nov 01 05:27:43.800797 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:43.800767    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9729d894-945b-4d8b-8700-46d3552c2c8e-metrics-certs\") pod \"maas-api-e2e-extmod-b-c6e99c-bbf76fcb8-bbrnt\" (UID: \"9729d894-945b-4d8b-8700-46d3552c2c8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c6e99c-bbf76fcb8-bbrnt"
Nov 01 05:27:43.800966 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:43.800861    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fwp7\" (UniqueName: \"kubernetes.io/projected/9729d894-945b-4d8b-8700-46d3552c2c8e-kube-api-access-6fwp7\") pod \"maas-api-e2e-extmod-b-c6e99c-bbf76fcb8-bbrnt\" (UID: \"9729d894-945b-4d8b-8700-46d3552c2c8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c6e99c-bbf76fcb8-bbrnt"
Nov 01 05:27:43.800966 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:43.800897    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9729d894-945b-4d8b-8700-46d3552c2c8e-maas-api-tls\") pod \"maas-api-e2e-extmod-b-c6e99c-bbf76fcb8-bbrnt\" (UID: \"9729d894-945b-4d8b-8700-46d3552c2c8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c6e99c-bbf76fcb8-bbrnt"
Nov 01 05:27:43.801080 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:43.801010    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9729d894-945b-4d8b-8700-46d3552c2c8e-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-c6e99c-bbf76fcb8-bbrnt\" (UID: \"9729d894-945b-4d8b-8700-46d3552c2c8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c6e99c-bbf76fcb8-bbrnt"
Nov 01 05:27:43.801138 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:27:43.801083    5689 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-c6e99c-serving-cert: secret "maas-api-e2e-extmod-b-c6e99c-serving-cert" not found
Nov 01 05:27:43.801190 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:27:43.801171    5689 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9729d894-945b-4d8b-8700-46d3552c2c8e-maas-api-tls podName:9729d894-945b-4d8b-8700-46d3552c2c8e nodeName:}" failed. No retries permitted until 2026-11-01 05:27:44.30115114 +0000 UTC m=+1442.371947856 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9729d894-945b-4d8b-8700-46d3552c2c8e-maas-api-tls") pod "maas-api-e2e-extmod-b-c6e99c-bbf76fcb8-bbrnt" (UID: "9729d894-945b-4d8b-8700-46d3552c2c8e") : secret "maas-api-e2e-extmod-b-c6e99c-serving-cert" not found
Nov 01 05:27:43.801622 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:43.801604    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9729d894-945b-4d8b-8700-46d3552c2c8e-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-c6e99c-bbf76fcb8-bbrnt\" (UID: \"9729d894-945b-4d8b-8700-46d3552c2c8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c6e99c-bbf76fcb8-bbrnt"
Nov 01 05:27:43.803365 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:43.803345    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9729d894-945b-4d8b-8700-46d3552c2c8e-metrics-certs\") pod \"maas-api-e2e-extmod-b-c6e99c-bbf76fcb8-bbrnt\" (UID: \"9729d894-945b-4d8b-8700-46d3552c2c8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c6e99c-bbf76fcb8-bbrnt"
Nov 01 05:27:43.862101 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:43.862077    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fwp7\" (UniqueName: \"kubernetes.io/projected/9729d894-945b-4d8b-8700-46d3552c2c8e-kube-api-access-6fwp7\") pod \"maas-api-e2e-extmod-b-c6e99c-bbf76fcb8-bbrnt\" (UID: \"9729d894-945b-4d8b-8700-46d3552c2c8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c6e99c-bbf76fcb8-bbrnt"
Nov 01 05:27:43.863695 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:43.863674    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:27:44.074226 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:44.074150    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:27:44.309543 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:44.309493    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9729d894-945b-4d8b-8700-46d3552c2c8e-maas-api-tls\") pod \"maas-api-e2e-extmod-b-c6e99c-bbf76fcb8-bbrnt\" (UID: \"9729d894-945b-4d8b-8700-46d3552c2c8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c6e99c-bbf76fcb8-bbrnt"
Nov 01 05:27:44.311949 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:44.311921    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9729d894-945b-4d8b-8700-46d3552c2c8e-maas-api-tls\") pod \"maas-api-e2e-extmod-b-c6e99c-bbf76fcb8-bbrnt\" (UID: \"9729d894-945b-4d8b-8700-46d3552c2c8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c6e99c-bbf76fcb8-bbrnt"
Nov 01 05:27:44.315140 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:44.315103    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:27:44.553010 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:44.552983    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c6e99c-bbf76fcb8-bbrnt"
Nov 01 05:27:44.934968 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:44.934947    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-c6e99c-bbf76fcb8-bbrnt"]
Nov 01 05:27:44.937054 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:27:44.937027    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9729d894_945b_4d8b_8700_46d3552c2c8e.slice/crio-2d006b3081dfcf86a2dedddee85c543d9b0ea32aa8bec24b379354cbf6412917 WatchSource:0}: Error finding container 2d006b3081dfcf86a2dedddee85c543d9b0ea32aa8bec24b379354cbf6412917: Status 404 returned error can't find the container with id 2d006b3081dfcf86a2dedddee85c543d9b0ea32aa8bec24b379354cbf6412917
Nov 01 05:27:44.938544 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:44.938527    5689 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 05:27:45.410285 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:45.410249    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c6e99c-bbf76fcb8-bbrnt" event={"ID":"9729d894-945b-4d8b-8700-46d3552c2c8e","Type":"ContainerStarted","Data":"2d006b3081dfcf86a2dedddee85c543d9b0ea32aa8bec24b379354cbf6412917"}
Nov 01 05:27:46.417073 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:46.417035    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c6e99c-bbf76fcb8-bbrnt" event={"ID":"9729d894-945b-4d8b-8700-46d3552c2c8e","Type":"ContainerStarted","Data":"788ea62d7ba6f99ef9e2900c01c9a1bdb67362c0b4ffc768bb4643dfb5efd0a5"}
Nov 01 05:27:46.417592 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:46.417095    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c6e99c-bbf76fcb8-bbrnt"
Nov 01 05:27:46.538011 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:46.537957    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c6e99c-bbf76fcb8-bbrnt" podStartSLOduration=2.652812337 podStartE2EDuration="3.537944547s" totalImagesPullingTime="885.13221ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:27:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:27:46.535632106 +0000 UTC m=+1444.606428827" watchObservedRunningTime="2026-11-01 05:27:46.537944547 +0000 UTC m=+1444.608741244"
Nov 01 05:27:48.300722 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:48.300687    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:27:49.182057 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:49.182020    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:27:52.428957 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:52.428879    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c6e99c-bbf76fcb8-bbrnt"
Nov 01 05:27:54.517476 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:54.515510    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:27:54.662836 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:54.662807    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:27:58.885620 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:58.885581    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:27:59.367953 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:27:59.367908    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:28:00.069345 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:28:00.069315    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:28:01.877061 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:01.877022    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:28:02.051990 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:02.051960    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:28:04.219599 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:04.219556    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:28:09.053947 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:09.053907    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:28:13.165692 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:13.165656    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:28:13.538351 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:13.538245    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5451cb-fdbf7d7f6-qwfh5"]
Nov 01 05:28:13.593013 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:13.592983    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5451cb-6cff848b7f-f7psh"]
Nov 01 05:28:13.593247 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:13.593202    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5451cb-6cff848b7f-f7psh" podUID="f4f05151-c697-49d3-bc36-f3e071865802" containerName="payload-pre-processing" containerID="cri-o://ebaf128572491b1eeb26c1265735a8f022f663f2ff86e27c286e5261f36470b8" gracePeriod=30
Nov 01 05:28:13.620491 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:13.620429    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-5451cb-fdbf7d7f6-qwfh5" podUID="cdb18db8-1611-478c-a170-26c41b29f020" containerName="payload-processing" containerID="cri-o://5272f36dc024b9ce2913b945d7e43d012eae0a91087738f0910e26611c587382" gracePeriod=30
Nov 01 05:28:13.726135 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:13.726082    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:28:13.905072 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:13.905041    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:28:13.909131 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:13.909110    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5451cb-6cff848b7f-f7psh"
Nov 01 05:28:13.912754 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:13.912735    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5451cb-fdbf7d7f6-qwfh5"
Nov 01 05:28:14.009191 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:14.009163    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdb18db8-1611-478c-a170-26c41b29f020-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdb18db8-1611-478c-a170-26c41b29f020-plugins-config-volume\") pod \"cdb18db8-1611-478c-a170-26c41b29f020\" (UID: \"cdb18db8-1611-478c-a170-26c41b29f020\") "
Nov 01 05:28:14.009352 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:14.009209    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4f05151-c697-49d3-bc36-f3e071865802-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4f05151-c697-49d3-bc36-f3e071865802-plugins-config-volume\") pod \"f4f05151-c697-49d3-bc36-f3e071865802\" (UID: \"f4f05151-c697-49d3-bc36-f3e071865802\") "
Nov 01 05:28:14.009352 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:14.009292    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4f05151-c697-49d3-bc36-f3e071865802-kube-api-access-mswds\" (UniqueName: \"kubernetes.io/projected/f4f05151-c697-49d3-bc36-f3e071865802-kube-api-access-mswds\") pod \"f4f05151-c697-49d3-bc36-f3e071865802\" (UID: \"f4f05151-c697-49d3-bc36-f3e071865802\") "
Nov 01 05:28:14.009525 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:14.009393    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdb18db8-1611-478c-a170-26c41b29f020-kube-api-access-k9jqk\" (UniqueName: \"kubernetes.io/projected/cdb18db8-1611-478c-a170-26c41b29f020-kube-api-access-k9jqk\") pod \"cdb18db8-1611-478c-a170-26c41b29f020\" (UID: \"cdb18db8-1611-478c-a170-26c41b29f020\") "
Nov 01 05:28:14.009670 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:14.009647    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdb18db8-1611-478c-a170-26c41b29f020-plugins-config-volume" pod "cdb18db8-1611-478c-a170-26c41b29f020" (UID: "cdb18db8-1611-478c-a170-26c41b29f020"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:28:14.009738 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:14.009664    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4f05151-c697-49d3-bc36-f3e071865802-plugins-config-volume" pod "f4f05151-c697-49d3-bc36-f3e071865802" (UID: "f4f05151-c697-49d3-bc36-f3e071865802"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:28:14.009918 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:14.009891    5689 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdb18db8-1611-478c-a170-26c41b29f020-plugins-config-volume\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:28:14.009918 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:14.009919    5689 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4f05151-c697-49d3-bc36-f3e071865802-plugins-config-volume\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:28:14.011598 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:14.011577    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4f05151-c697-49d3-bc36-f3e071865802-kube-api-access-mswds" pod "f4f05151-c697-49d3-bc36-f3e071865802" (UID: "f4f05151-c697-49d3-bc36-f3e071865802"). InnerVolumeSpecName "kube-api-access-mswds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:28:14.011677 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:14.011651    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdb18db8-1611-478c-a170-26c41b29f020-kube-api-access-k9jqk" pod "cdb18db8-1611-478c-a170-26c41b29f020" (UID: "cdb18db8-1611-478c-a170-26c41b29f020"). InnerVolumeSpecName "kube-api-access-k9jqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:28:14.113062 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:14.112977    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mswds\" (UniqueName: \"kubernetes.io/projected/f4f05151-c697-49d3-bc36-f3e071865802-kube-api-access-mswds\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:28:14.113343 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:14.113322    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9jqk\" (UniqueName: \"kubernetes.io/projected/cdb18db8-1611-478c-a170-26c41b29f020-kube-api-access-k9jqk\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:28:14.626947 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:14.626917    5689 generic.go:356] "Generic (PLEG): container finished" podID="cdb18db8-1611-478c-a170-26c41b29f020" containerID="5272f36dc024b9ce2913b945d7e43d012eae0a91087738f0910e26611c587382" exitCode=0
Nov 01 05:28:14.627410 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:14.626966    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5451cb-fdbf7d7f6-qwfh5"
Nov 01 05:28:14.627410 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:14.627008    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5451cb-fdbf7d7f6-qwfh5" event={"ID":"cdb18db8-1611-478c-a170-26c41b29f020","Type":"ContainerDied","Data":"5272f36dc024b9ce2913b945d7e43d012eae0a91087738f0910e26611c587382"}
Nov 01 05:28:14.627410 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:14.627058    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5451cb-fdbf7d7f6-qwfh5" event={"ID":"cdb18db8-1611-478c-a170-26c41b29f020","Type":"ContainerDied","Data":"c7323caec5569925c014cfa0f0d1c6b5ca17025a0f82cb4ef43175cbd48586c3"}
Nov 01 05:28:14.627410 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:14.627085    5689 scope.go:122] "RemoveContainer" containerID="5272f36dc024b9ce2913b945d7e43d012eae0a91087738f0910e26611c587382"
Nov 01 05:28:14.628585 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:14.628561    5689 generic.go:356] "Generic (PLEG): container finished" podID="f4f05151-c697-49d3-bc36-f3e071865802" containerID="ebaf128572491b1eeb26c1265735a8f022f663f2ff86e27c286e5261f36470b8" exitCode=0
Nov 01 05:28:14.628854 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:14.628598    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5451cb-6cff848b7f-f7psh"
Nov 01 05:28:14.628854 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:14.628676    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5451cb-6cff848b7f-f7psh" event={"ID":"f4f05151-c697-49d3-bc36-f3e071865802","Type":"ContainerDied","Data":"ebaf128572491b1eeb26c1265735a8f022f663f2ff86e27c286e5261f36470b8"}
Nov 01 05:28:14.628854 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:14.628713    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5451cb-6cff848b7f-f7psh" event={"ID":"f4f05151-c697-49d3-bc36-f3e071865802","Type":"ContainerDied","Data":"4ccd9dd84b75367f2c39c5484583b6ef69ad2b9d9e1bc8665afc072783e5dfdd"}
Nov 01 05:28:14.637064 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:14.637044    5689 scope.go:122] "RemoveContainer" containerID="5272f36dc024b9ce2913b945d7e43d012eae0a91087738f0910e26611c587382"
Nov 01 05:28:14.637312 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:28:14.637294    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5272f36dc024b9ce2913b945d7e43d012eae0a91087738f0910e26611c587382\": container with ID starting with 5272f36dc024b9ce2913b945d7e43d012eae0a91087738f0910e26611c587382 not found: ID does not exist" containerID="5272f36dc024b9ce2913b945d7e43d012eae0a91087738f0910e26611c587382"
Nov 01 05:28:14.637357 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:14.637322    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5272f36dc024b9ce2913b945d7e43d012eae0a91087738f0910e26611c587382"} err="failed to get container status \"5272f36dc024b9ce2913b945d7e43d012eae0a91087738f0910e26611c587382\": rpc error: code = NotFound desc = could not find container \"5272f36dc024b9ce2913b945d7e43d012eae0a91087738f0910e26611c587382\": container with ID starting with 5272f36dc024b9ce2913b945d7e43d012eae0a91087738f0910e26611c587382 not found: ID does not exist"
Nov 01 05:28:14.637357 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:14.637352    5689 scope.go:122] "RemoveContainer" containerID="ebaf128572491b1eeb26c1265735a8f022f663f2ff86e27c286e5261f36470b8"
Nov 01 05:28:14.645702 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:14.645685    5689 scope.go:122] "RemoveContainer" containerID="ebaf128572491b1eeb26c1265735a8f022f663f2ff86e27c286e5261f36470b8"
Nov 01 05:28:14.645971 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:28:14.645956    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebaf128572491b1eeb26c1265735a8f022f663f2ff86e27c286e5261f36470b8\": container with ID starting with ebaf128572491b1eeb26c1265735a8f022f663f2ff86e27c286e5261f36470b8 not found: ID does not exist" containerID="ebaf128572491b1eeb26c1265735a8f022f663f2ff86e27c286e5261f36470b8"
Nov 01 05:28:14.646017 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:14.645979    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebaf128572491b1eeb26c1265735a8f022f663f2ff86e27c286e5261f36470b8"} err="failed to get container status \"ebaf128572491b1eeb26c1265735a8f022f663f2ff86e27c286e5261f36470b8\": rpc error: code = NotFound desc = could not find container \"ebaf128572491b1eeb26c1265735a8f022f663f2ff86e27c286e5261f36470b8\": container with ID starting with ebaf128572491b1eeb26c1265735a8f022f663f2ff86e27c286e5261f36470b8 not found: ID does not exist"
Nov 01 05:28:14.790785 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:14.790749    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5451cb-6cff848b7f-f7psh"]
Nov 01 05:28:14.798835 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:14.798809    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5451cb-6cff848b7f-f7psh"]
Nov 01 05:28:14.906638 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:14.906614    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5451cb-fdbf7d7f6-qwfh5"]
Nov 01 05:28:14.910776 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:14.910753    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5451cb-fdbf7d7f6-qwfh5"]
Nov 01 05:28:15.050954 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:15.050916    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:28:16.068300 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:16.068268    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdb18db8-1611-478c-a170-26c41b29f020" path="/var/lib/kubelet/pods/cdb18db8-1611-478c-a170-26c41b29f020/volumes"
Nov 01 05:28:16.068839 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:16.068815    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4f05151-c697-49d3-bc36-f3e071865802" path="/var/lib/kubelet/pods/f4f05151-c697-49d3-bc36-f3e071865802/volumes"
Nov 01 05:28:16.069954 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:16.069933    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:28:19.895672 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:19.895614    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:28:20.320418 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:20.320377    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:28:20.495261 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:20.495218    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:28:21.913361 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:21.913311    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:28:22.409853 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:22.409826    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd"]
Nov 01 05:28:22.410029 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:22.410011    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd" podUID="a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50" containerName="istio-proxy" containerID="cri-o://765d1c81e31cc0bfb6a85bba43660d16aad48259c7a496acfe3fcdf60fe38e91" gracePeriod=30
Nov 01 05:28:22.440352 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:22.440300    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:28:25.128642 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:25.128610    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:28:27.669830 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:27.669806    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd"
Nov 01 05:28:27.718566 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:27.718513    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50-istio-token\" (UniqueName: \"kubernetes.io/projected/a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50-istio-token\") pod \"a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50\" (UID: \"a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50\") "
Nov 01 05:28:27.718566 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:27.718543    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50-istiod-ca-cert\") pod \"a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50\" (UID: \"a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50\") "
Nov 01 05:28:27.718741 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:27.718648    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50-istio-data\") pod \"a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50\" (UID: \"a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50\") "
Nov 01 05:28:27.718741 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:27.718683    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50-workload-socket\") pod \"a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50\" (UID: \"a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50\") "
Nov 01 05:28:27.718741 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:27.718723    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50-workload-certs\") pod \"a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50\" (UID: \"a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50\") "
Nov 01 05:28:27.718903 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:27.718779    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50-istio-envoy\") pod \"a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50\" (UID: \"a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50\") "
Nov 01 05:28:27.718903 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:27.718851    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50-kube-api-access-zdm75\" (UniqueName: \"kubernetes.io/projected/a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50-kube-api-access-zdm75\") pod \"a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50\" (UID: \"a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50\") "
Nov 01 05:28:27.718903 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:27.718894    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50-istio-podinfo\") pod \"a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50\" (UID: \"a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50\") "
Nov 01 05:28:27.719053 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:27.718965    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50-credential-socket\") pod \"a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50\" (UID: \"a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50\") "
Nov 01 05:28:27.719053 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:27.718949    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50-workload-socket" pod "a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50" (UID: "a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:28:27.719053 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:27.718977    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50-istio-data" pod "a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50" (UID: "a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:28:27.719053 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:27.719023    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50-workload-certs" pod "a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50" (UID: "a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:28:27.719249 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:27.719054    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50-istiod-ca-cert" pod "a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50" (UID: "a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:28:27.719307 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:27.719281    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50-credential-socket" pod "a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50" (UID: "a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:28:27.719703 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:27.719619    5689 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50-istiod-ca-cert\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:28:27.719703 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:27.719644    5689 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50-istio-data\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:28:27.719703 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:27.719653    5689 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50-workload-socket\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:28:27.719703 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:27.719662    5689 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50-workload-certs\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:28:27.719703 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:27.719670    5689 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50-credential-socket\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:28:27.721096 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:27.721066    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50-istio-envoy" pod "a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50" (UID: "a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:28:27.721208 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:27.721183    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50-istio-token" pod "a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50" (UID: "a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:28:27.721542 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:27.721520    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50-istio-podinfo" pod "a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50" (UID: "a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 05:28:27.721646 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:27.721627    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50-kube-api-access-zdm75" pod "a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50" (UID: "a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50"). InnerVolumeSpecName "kube-api-access-zdm75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:28:27.732261 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:27.732236    5689 generic.go:356] "Generic (PLEG): container finished" podID="a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50" containerID="765d1c81e31cc0bfb6a85bba43660d16aad48259c7a496acfe3fcdf60fe38e91" exitCode=0
Nov 01 05:28:27.732358 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:27.732288    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd"
Nov 01 05:28:27.732358 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:27.732295    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd" event={"ID":"a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50","Type":"ContainerDied","Data":"765d1c81e31cc0bfb6a85bba43660d16aad48259c7a496acfe3fcdf60fe38e91"}
Nov 01 05:28:27.732501 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:27.732366    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd" event={"ID":"a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50","Type":"ContainerDied","Data":"16768705388d8a6a3b95f6b52888c22ad3712b4f0858c8799f14c6c554b19bf0"}
Nov 01 05:28:27.732501 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:27.732389    5689 scope.go:122] "RemoveContainer" containerID="765d1c81e31cc0bfb6a85bba43660d16aad48259c7a496acfe3fcdf60fe38e91"
Nov 01 05:28:27.745472 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:27.745440    5689 scope.go:122] "RemoveContainer" containerID="e471ff50c9e2245d141f675eba5be657e2eab2385dfd2d032238ae783a61319e"
Nov 01 05:28:27.753885 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:27.753869    5689 scope.go:122] "RemoveContainer" containerID="765d1c81e31cc0bfb6a85bba43660d16aad48259c7a496acfe3fcdf60fe38e91"
Nov 01 05:28:27.754125 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:28:27.754108    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"765d1c81e31cc0bfb6a85bba43660d16aad48259c7a496acfe3fcdf60fe38e91\": container with ID starting with 765d1c81e31cc0bfb6a85bba43660d16aad48259c7a496acfe3fcdf60fe38e91 not found: ID does not exist" containerID="765d1c81e31cc0bfb6a85bba43660d16aad48259c7a496acfe3fcdf60fe38e91"
Nov 01 05:28:27.754169 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:27.754132    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"765d1c81e31cc0bfb6a85bba43660d16aad48259c7a496acfe3fcdf60fe38e91"} err="failed to get container status \"765d1c81e31cc0bfb6a85bba43660d16aad48259c7a496acfe3fcdf60fe38e91\": rpc error: code = NotFound desc = could not find container \"765d1c81e31cc0bfb6a85bba43660d16aad48259c7a496acfe3fcdf60fe38e91\": container with ID starting with 765d1c81e31cc0bfb6a85bba43660d16aad48259c7a496acfe3fcdf60fe38e91 not found: ID does not exist"
Nov 01 05:28:27.754169 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:27.754148    5689 scope.go:122] "RemoveContainer" containerID="e471ff50c9e2245d141f675eba5be657e2eab2385dfd2d032238ae783a61319e"
Nov 01 05:28:27.754376 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:28:27.754357    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e471ff50c9e2245d141f675eba5be657e2eab2385dfd2d032238ae783a61319e\": container with ID starting with e471ff50c9e2245d141f675eba5be657e2eab2385dfd2d032238ae783a61319e not found: ID does not exist" containerID="e471ff50c9e2245d141f675eba5be657e2eab2385dfd2d032238ae783a61319e"
Nov 01 05:28:27.754413 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:27.754383    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e471ff50c9e2245d141f675eba5be657e2eab2385dfd2d032238ae783a61319e"} err="failed to get container status \"e471ff50c9e2245d141f675eba5be657e2eab2385dfd2d032238ae783a61319e\": rpc error: code = NotFound desc = could not find container \"e471ff50c9e2245d141f675eba5be657e2eab2385dfd2d032238ae783a61319e\": container with ID starting with e471ff50c9e2245d141f675eba5be657e2eab2385dfd2d032238ae783a61319e not found: ID does not exist"
Nov 01 05:28:27.820888 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:27.820863    5689 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50-istio-envoy\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:28:27.820888 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:27.820886    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdm75\" (UniqueName: \"kubernetes.io/projected/a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50-kube-api-access-zdm75\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:28:27.821033 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:27.820897    5689 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50-istio-podinfo\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:28:27.821033 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:27.820912    5689 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50-istio-token\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:28:27.949979 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:27.949946    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd"]
Nov 01 05:28:27.954510 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:27.954481    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-5451cb-openshift-default-5965d59997-pgsbd"]
Nov 01 05:28:28.067423 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:28.067347    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50" path="/var/lib/kubelet/pods/a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50/volumes"
Nov 01 05:28:30.007117 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:30.007074    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:28:30.110315 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:28:30.110279    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:28:34.801848 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:34.801816    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:28:39.612253 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:39.612219    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:28:41.763328 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:41.763293    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-7d1ec3/distinct-w0-7d1ec3-kserve-58cdfc8d58-n7xpt"]
Nov 01 05:28:41.763871 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:41.763855    5689 cpu_manager.go:414] "containerMap: removing container" podUID="a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50" containerName="istio-proxy"
Nov 01 05:28:41.763931 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:41.763872    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50" containerName="istio-proxy"
Nov 01 05:28:41.763931 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:41.763886    5689 cpu_manager.go:414] "containerMap: removing container" podUID="f4f05151-c697-49d3-bc36-f3e071865802" containerName="payload-pre-processing"
Nov 01 05:28:41.763931 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:41.763892    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4f05151-c697-49d3-bc36-f3e071865802" containerName="payload-pre-processing"
Nov 01 05:28:41.763931 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:41.763912    5689 cpu_manager.go:414] "containerMap: removing container" podUID="cdb18db8-1611-478c-a170-26c41b29f020" containerName="payload-processing"
Nov 01 05:28:41.763931 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:41.763919    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdb18db8-1611-478c-a170-26c41b29f020" containerName="payload-processing"
Nov 01 05:28:41.764077 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:41.764007    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdb18db8-1611-478c-a170-26c41b29f020" containerName="payload-processing"
Nov 01 05:28:41.764077 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:41.764019    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50" containerName="istio-proxy"
Nov 01 05:28:41.764077 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:41.764026    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50" containerName="istio-proxy"
Nov 01 05:28:41.764077 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:41.764031    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4f05151-c697-49d3-bc36-f3e071865802" containerName="payload-pre-processing"
Nov 01 05:28:41.767392 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:41.767377    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7d1ec3/distinct-w0-7d1ec3-kserve-58cdfc8d58-n7xpt"
Nov 01 05:28:41.784099 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:41.784048    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:28:41.876004 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:41.875972    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a1fcaaf-0b04-4602-8985-866f6152303d-tmp-dir\") pod \"distinct-w0-7d1ec3-kserve-58cdfc8d58-n7xpt\" (UID: \"4a1fcaaf-0b04-4602-8985-866f6152303d\") " pod="e2e-models-e2e-worker-w0-7d1ec3/distinct-w0-7d1ec3-kserve-58cdfc8d58-n7xpt"
Nov 01 05:28:41.876004 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:41.876010    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a1fcaaf-0b04-4602-8985-866f6152303d-home\") pod \"distinct-w0-7d1ec3-kserve-58cdfc8d58-n7xpt\" (UID: \"4a1fcaaf-0b04-4602-8985-866f6152303d\") " pod="e2e-models-e2e-worker-w0-7d1ec3/distinct-w0-7d1ec3-kserve-58cdfc8d58-n7xpt"
Nov 01 05:28:41.876253 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:41.876096    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a1fcaaf-0b04-4602-8985-866f6152303d-dshm\") pod \"distinct-w0-7d1ec3-kserve-58cdfc8d58-n7xpt\" (UID: \"4a1fcaaf-0b04-4602-8985-866f6152303d\") " pod="e2e-models-e2e-worker-w0-7d1ec3/distinct-w0-7d1ec3-kserve-58cdfc8d58-n7xpt"
Nov 01 05:28:41.876253 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:41.876146    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a1fcaaf-0b04-4602-8985-866f6152303d-tls-certs\") pod \"distinct-w0-7d1ec3-kserve-58cdfc8d58-n7xpt\" (UID: \"4a1fcaaf-0b04-4602-8985-866f6152303d\") " pod="e2e-models-e2e-worker-w0-7d1ec3/distinct-w0-7d1ec3-kserve-58cdfc8d58-n7xpt"
Nov 01 05:28:41.876359 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:41.876255    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4tp7\" (UniqueName: \"kubernetes.io/projected/4a1fcaaf-0b04-4602-8985-866f6152303d-kube-api-access-j4tp7\") pod \"distinct-w0-7d1ec3-kserve-58cdfc8d58-n7xpt\" (UID: \"4a1fcaaf-0b04-4602-8985-866f6152303d\") " pod="e2e-models-e2e-worker-w0-7d1ec3/distinct-w0-7d1ec3-kserve-58cdfc8d58-n7xpt"
Nov 01 05:28:41.876359 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:41.876326    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a1fcaaf-0b04-4602-8985-866f6152303d-model-cache\") pod \"distinct-w0-7d1ec3-kserve-58cdfc8d58-n7xpt\" (UID: \"4a1fcaaf-0b04-4602-8985-866f6152303d\") " pod="e2e-models-e2e-worker-w0-7d1ec3/distinct-w0-7d1ec3-kserve-58cdfc8d58-n7xpt"
Nov 01 05:28:41.897363 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:41.897339    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-7d1ec3\"/\"kube-root-ca.crt\""
Nov 01 05:28:41.897538 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:41.897379    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-7d1ec3\"/\"distinct-w0-7d1ec3-kserve-self-signed-certs\""
Nov 01 05:28:41.897538 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:41.897380    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-7d1ec3\"/\"openshift-service-ca.crt\""
Nov 01 05:28:41.897538 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:41.897397    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-7d1ec3\"/\"default-dockercfg-4l4n6\""
Nov 01 05:28:41.899920 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:41.899897    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7d1ec3/distinct-w0-7d1ec3-kserve-58cdfc8d58-n7xpt"]
Nov 01 05:28:41.985062 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:41.977191    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a1fcaaf-0b04-4602-8985-866f6152303d-dshm\") pod \"distinct-w0-7d1ec3-kserve-58cdfc8d58-n7xpt\" (UID: \"4a1fcaaf-0b04-4602-8985-866f6152303d\") " pod="e2e-models-e2e-worker-w0-7d1ec3/distinct-w0-7d1ec3-kserve-58cdfc8d58-n7xpt"
Nov 01 05:28:41.985062 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:41.977272    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a1fcaaf-0b04-4602-8985-866f6152303d-tls-certs\") pod \"distinct-w0-7d1ec3-kserve-58cdfc8d58-n7xpt\" (UID: \"4a1fcaaf-0b04-4602-8985-866f6152303d\") " pod="e2e-models-e2e-worker-w0-7d1ec3/distinct-w0-7d1ec3-kserve-58cdfc8d58-n7xpt"
Nov 01 05:28:41.985062 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:41.977351    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4tp7\" (UniqueName: \"kubernetes.io/projected/4a1fcaaf-0b04-4602-8985-866f6152303d-kube-api-access-j4tp7\") pod \"distinct-w0-7d1ec3-kserve-58cdfc8d58-n7xpt\" (UID: \"4a1fcaaf-0b04-4602-8985-866f6152303d\") " pod="e2e-models-e2e-worker-w0-7d1ec3/distinct-w0-7d1ec3-kserve-58cdfc8d58-n7xpt"
Nov 01 05:28:41.985062 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:41.977412    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a1fcaaf-0b04-4602-8985-866f6152303d-model-cache\") pod \"distinct-w0-7d1ec3-kserve-58cdfc8d58-n7xpt\" (UID: \"4a1fcaaf-0b04-4602-8985-866f6152303d\") " pod="e2e-models-e2e-worker-w0-7d1ec3/distinct-w0-7d1ec3-kserve-58cdfc8d58-n7xpt"
Nov 01 05:28:41.985062 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:41.977589    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a1fcaaf-0b04-4602-8985-866f6152303d-tmp-dir\") pod \"distinct-w0-7d1ec3-kserve-58cdfc8d58-n7xpt\" (UID: \"4a1fcaaf-0b04-4602-8985-866f6152303d\") " pod="e2e-models-e2e-worker-w0-7d1ec3/distinct-w0-7d1ec3-kserve-58cdfc8d58-n7xpt"
Nov 01 05:28:41.985062 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:41.977621    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a1fcaaf-0b04-4602-8985-866f6152303d-home\") pod \"distinct-w0-7d1ec3-kserve-58cdfc8d58-n7xpt\" (UID: \"4a1fcaaf-0b04-4602-8985-866f6152303d\") " pod="e2e-models-e2e-worker-w0-7d1ec3/distinct-w0-7d1ec3-kserve-58cdfc8d58-n7xpt"
Nov 01 05:28:41.985062 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:41.978121    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a1fcaaf-0b04-4602-8985-866f6152303d-home\") pod \"distinct-w0-7d1ec3-kserve-58cdfc8d58-n7xpt\" (UID: \"4a1fcaaf-0b04-4602-8985-866f6152303d\") " pod="e2e-models-e2e-worker-w0-7d1ec3/distinct-w0-7d1ec3-kserve-58cdfc8d58-n7xpt"
Nov 01 05:28:41.988846 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:41.988814    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a1fcaaf-0b04-4602-8985-866f6152303d-model-cache\") pod \"distinct-w0-7d1ec3-kserve-58cdfc8d58-n7xpt\" (UID: \"4a1fcaaf-0b04-4602-8985-866f6152303d\") " pod="e2e-models-e2e-worker-w0-7d1ec3/distinct-w0-7d1ec3-kserve-58cdfc8d58-n7xpt"
Nov 01 05:28:41.989309 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:41.989284    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a1fcaaf-0b04-4602-8985-866f6152303d-tmp-dir\") pod \"distinct-w0-7d1ec3-kserve-58cdfc8d58-n7xpt\" (UID: \"4a1fcaaf-0b04-4602-8985-866f6152303d\") " pod="e2e-models-e2e-worker-w0-7d1ec3/distinct-w0-7d1ec3-kserve-58cdfc8d58-n7xpt"
Nov 01 05:28:41.994826 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:41.989757    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:28:42.000182 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:42.000156    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a1fcaaf-0b04-4602-8985-866f6152303d-tls-certs\") pod \"distinct-w0-7d1ec3-kserve-58cdfc8d58-n7xpt\" (UID: \"4a1fcaaf-0b04-4602-8985-866f6152303d\") " pod="e2e-models-e2e-worker-w0-7d1ec3/distinct-w0-7d1ec3-kserve-58cdfc8d58-n7xpt"
Nov 01 05:28:42.011727 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:42.011693    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a1fcaaf-0b04-4602-8985-866f6152303d-dshm\") pod \"distinct-w0-7d1ec3-kserve-58cdfc8d58-n7xpt\" (UID: \"4a1fcaaf-0b04-4602-8985-866f6152303d\") " pod="e2e-models-e2e-worker-w0-7d1ec3/distinct-w0-7d1ec3-kserve-58cdfc8d58-n7xpt"
Nov 01 05:28:42.094527 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:42.094430    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-7d1ec3\"/\"kube-root-ca.crt\""
Nov 01 05:28:42.156967 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:42.156938    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-7d1ec3\"/\"openshift-service-ca.crt\""
Nov 01 05:28:42.182500 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:42.177901    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4tp7\" (UniqueName: \"kubernetes.io/projected/4a1fcaaf-0b04-4602-8985-866f6152303d-kube-api-access-j4tp7\") pod \"distinct-w0-7d1ec3-kserve-58cdfc8d58-n7xpt\" (UID: \"4a1fcaaf-0b04-4602-8985-866f6152303d\") " pod="e2e-models-e2e-worker-w0-7d1ec3/distinct-w0-7d1ec3-kserve-58cdfc8d58-n7xpt"
Nov 01 05:28:42.430828 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:42.430801    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-7d1ec3\"/\"default-dockercfg-4l4n6\""
Nov 01 05:28:42.440056 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:42.440029    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7d1ec3/distinct-w0-7d1ec3-kserve-58cdfc8d58-n7xpt"
Nov 01 05:28:42.819401 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:42.818961    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7d1ec3/distinct-w0-7d1ec3-kserve-58cdfc8d58-n7xpt"]
Nov 01 05:28:42.872517 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:42.872486    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7d1ec3/distinct-w0-7d1ec3-kserve-58cdfc8d58-n7xpt" event={"ID":"4a1fcaaf-0b04-4602-8985-866f6152303d","Type":"ContainerStarted","Data":"8f762389e9cc8fea55cf391276ba50acb1ea3b57a034843ce937025b5b494cd6"}
Nov 01 05:28:43.000398 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:43.000369    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:28:43.177089 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:43.177062    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:28:43.365403 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:43.365371    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:28:43.644561 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:43.644384    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:28:43.879432 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:43.879399    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7d1ec3/distinct-w0-7d1ec3-kserve-58cdfc8d58-n7xpt" event={"ID":"4a1fcaaf-0b04-4602-8985-866f6152303d","Type":"ContainerStarted","Data":"231e59b47cff2533f6dc91b77dd4158b636975474c6867bbae7cf753f168693c"}
Nov 01 05:28:43.956351 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:43.956326    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:28:43.997012 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:43.996956    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-7d1ec3/distinct-w0-7d1ec3-kserve-58cdfc8d58-n7xpt" podStartSLOduration=2.996943608 podStartE2EDuration="2.996943608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:28:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:28:43.994578836 +0000 UTC m=+1502.065375549" watchObservedRunningTime="2026-11-01 05:28:43.996943608 +0000 UTC m=+1502.067740319"
Nov 01 05:28:44.136476 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:44.135967    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:28:44.440252 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:44.440210    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7d1ec3/distinct-w0-7d1ec3-kserve-58cdfc8d58-n7xpt"
Nov 01 05:28:44.457146 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:44.457111    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:28:44.657925 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:44.657889    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:28:44.839367 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:44.839287    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:28:45.020586 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:45.020551    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:28:46.303232 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:46.303200    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:28:46.505906 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:46.505824    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:28:48.727762 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:48.727720    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:28:49.144009 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:49.143942    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-d989mcgpt"]
Nov 01 05:28:49.144507 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:49.144493    5689 cpu_manager.go:414] "containerMap: removing container" podUID="a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50" containerName="istio-proxy"
Nov 01 05:28:49.144555 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:49.144509    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a43ac5f8-e7ab-4e5d-a8a9-129926f1ba50" containerName="istio-proxy"
Nov 01 05:28:49.148033 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:49.148016    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-d989mcgpt"
Nov 01 05:28:49.224385 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:49.224347    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/494e7554-111b-4033-8a49-68ae0ad86c0b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-d989mcgpt\" (UID: \"494e7554-111b-4033-8a49-68ae0ad86c0b\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-d989mcgpt"
Nov 01 05:28:49.224534 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:49.224518    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tf5j\" (UniqueName: \"kubernetes.io/projected/494e7554-111b-4033-8a49-68ae0ad86c0b-kube-api-access-7tf5j\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-d989mcgpt\" (UID: \"494e7554-111b-4033-8a49-68ae0ad86c0b\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-d989mcgpt"
Nov 01 05:28:49.224648 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:49.224628    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/494e7554-111b-4033-8a49-68ae0ad86c0b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-d989mcgpt\" (UID: \"494e7554-111b-4033-8a49-68ae0ad86c0b\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-d989mcgpt"
Nov 01 05:28:49.224717 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:49.224678    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/494e7554-111b-4033-8a49-68ae0ad86c0b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-d989mcgpt\" (UID: \"494e7554-111b-4033-8a49-68ae0ad86c0b\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-d989mcgpt"
Nov 01 05:28:49.249610 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:49.249572    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c6e99c\"/\"routing-overlay\""
Nov 01 05:28:49.249610 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:49.249588    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-c6e99c\"/\"agc-cred-17c481ce95fa\""
Nov 01 05:28:49.249610 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:49.249606    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c6e99c\"/\"kube-root-ca.crt\""
Nov 01 05:28:49.249871 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:49.249611    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c6e99c\"/\"openshift-service-ca.crt\""
Nov 01 05:28:49.249871 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:49.249622    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c6e99c\"/\"payload-processing-external-model-plugins-e2e-extmod-b-c6e99c\""
Nov 01 05:28:49.252083 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:49.252059    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-d989mcgpt"]
Nov 01 05:28:49.325749 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:49.325722    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/494e7554-111b-4033-8a49-68ae0ad86c0b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-d989mcgpt\" (UID: \"494e7554-111b-4033-8a49-68ae0ad86c0b\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-d989mcgpt"
Nov 01 05:28:49.325904 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:49.325760    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/494e7554-111b-4033-8a49-68ae0ad86c0b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-d989mcgpt\" (UID: \"494e7554-111b-4033-8a49-68ae0ad86c0b\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-d989mcgpt"
Nov 01 05:28:49.325904 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:49.325866    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/494e7554-111b-4033-8a49-68ae0ad86c0b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-d989mcgpt\" (UID: \"494e7554-111b-4033-8a49-68ae0ad86c0b\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-d989mcgpt"
Nov 01 05:28:49.326026 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:49.325921    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tf5j\" (UniqueName: \"kubernetes.io/projected/494e7554-111b-4033-8a49-68ae0ad86c0b-kube-api-access-7tf5j\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-d989mcgpt\" (UID: \"494e7554-111b-4033-8a49-68ae0ad86c0b\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-d989mcgpt"
Nov 01 05:28:49.326415 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:49.326393    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/494e7554-111b-4033-8a49-68ae0ad86c0b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-d989mcgpt\" (UID: \"494e7554-111b-4033-8a49-68ae0ad86c0b\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-d989mcgpt"
Nov 01 05:28:49.326619 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:49.326602    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/494e7554-111b-4033-8a49-68ae0ad86c0b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-d989mcgpt\" (UID: \"494e7554-111b-4033-8a49-68ae0ad86c0b\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-d989mcgpt"
Nov 01 05:28:49.328361 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:49.328341    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/494e7554-111b-4033-8a49-68ae0ad86c0b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-d989mcgpt\" (UID: \"494e7554-111b-4033-8a49-68ae0ad86c0b\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-d989mcgpt"
Nov 01 05:28:49.386389 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:49.386364    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tf5j\" (UniqueName: \"kubernetes.io/projected/494e7554-111b-4033-8a49-68ae0ad86c0b-kube-api-access-7tf5j\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-d989mcgpt\" (UID: \"494e7554-111b-4033-8a49-68ae0ad86c0b\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-d989mcgpt"
Nov 01 05:28:49.463035 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:49.463013    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-d989mcgpt"
Nov 01 05:28:49.598599 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:49.598573    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:28:49.844562 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:49.844537    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-d989mcgpt"]
Nov 01 05:28:49.847519 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:28:49.847493    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod494e7554_111b_4033_8a49_68ae0ad86c0b.slice/crio-88323d7c20147730126c3c50ab58f12a85dcebccff10a7139a47044ab62152c9 WatchSource:0}: Error finding container 88323d7c20147730126c3c50ab58f12a85dcebccff10a7139a47044ab62152c9: Status 404 returned error can't find the container with id 88323d7c20147730126c3c50ab58f12a85dcebccff10a7139a47044ab62152c9
Nov 01 05:28:49.928357 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:49.928327    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-d989mcgpt" event={"ID":"494e7554-111b-4033-8a49-68ae0ad86c0b","Type":"ContainerStarted","Data":"22fae2f9c602d3da8b94d4c8ee15075b806608777b8079a03711867bd5ceaf67"}
Nov 01 05:28:49.928759 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:49.928393    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-d989mcgpt"
Nov 01 05:28:49.928759 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:49.928419    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-d989mcgpt" event={"ID":"494e7554-111b-4033-8a49-68ae0ad86c0b","Type":"ContainerStarted","Data":"88323d7c20147730126c3c50ab58f12a85dcebccff10a7139a47044ab62152c9"}
Nov 01 05:28:50.048563 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:50.048518    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-d989mcgpt" podStartSLOduration=1.048505604 podStartE2EDuration="1.048505604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:28:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:28:50.047763113 +0000 UTC m=+1508.118559833" watchObservedRunningTime="2026-11-01 05:28:50.048505604 +0000 UTC m=+1508.119302323"
Nov 01 05:28:51.263674 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:51.263638    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:28:52.378311 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:52.378276    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-9db8tm77q"]
Nov 01 05:28:52.382248 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:52.382232    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-9db8tm77q"
Nov 01 05:28:52.434774 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:52.434744    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-dee6aa\"/\"payload-processing-external-model-plugins-e2e-extmod-a-dee6aa\""
Nov 01 05:28:52.434909 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:52.434818    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-dee6aa\"/\"routing-overlay\""
Nov 01 05:28:52.440444 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:52.440414    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-7d1ec3/distinct-w0-7d1ec3-kserve-58cdfc8d58-n7xpt"
Nov 01 05:28:52.454731 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:52.454707    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-7d1ec3/distinct-w0-7d1ec3-kserve-58cdfc8d58-n7xpt"
Nov 01 05:28:52.483910 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:52.483888    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-dee6aa\"/\"openshift-service-ca.crt\""
Nov 01 05:28:52.484048 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:52.484008    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-dee6aa\"/\"kube-root-ca.crt\""
Nov 01 05:28:52.484048 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:52.484026    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-dee6aa\"/\"agc-cred-3bbeecafc00b\""
Nov 01 05:28:52.484168 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:52.484105    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-dee6aa\"/\"payload-processing-external-model-e2e--19b5d07b-dockercfg-89znj\""
Nov 01 05:28:52.486573 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:52.486557    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-9db8tm77q"]
Nov 01 05:28:52.518284 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:52.518255    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8c6e7e1d-dfc3-49a1-a6e0-acb6937eecde-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-9db8tm77q\" (UID: \"8c6e7e1d-dfc3-49a1-a6e0-acb6937eecde\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-9db8tm77q"
Nov 01 05:28:52.518408 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:52.518388    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c6e7e1d-dfc3-49a1-a6e0-acb6937eecde-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-9db8tm77q\" (UID: \"8c6e7e1d-dfc3-49a1-a6e0-acb6937eecde\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-9db8tm77q"
Nov 01 05:28:52.518520 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:52.518492    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8c6e7e1d-dfc3-49a1-a6e0-acb6937eecde-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-9db8tm77q\" (UID: \"8c6e7e1d-dfc3-49a1-a6e0-acb6937eecde\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-9db8tm77q"
Nov 01 05:28:52.518624 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:52.518609    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nz86\" (UniqueName: \"kubernetes.io/projected/8c6e7e1d-dfc3-49a1-a6e0-acb6937eecde-kube-api-access-5nz86\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-9db8tm77q\" (UID: \"8c6e7e1d-dfc3-49a1-a6e0-acb6937eecde\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-9db8tm77q"
Nov 01 05:28:52.619268 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:52.619238    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8c6e7e1d-dfc3-49a1-a6e0-acb6937eecde-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-9db8tm77q\" (UID: \"8c6e7e1d-dfc3-49a1-a6e0-acb6937eecde\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-9db8tm77q"
Nov 01 05:28:52.619397 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:52.619318    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c6e7e1d-dfc3-49a1-a6e0-acb6937eecde-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-9db8tm77q\" (UID: \"8c6e7e1d-dfc3-49a1-a6e0-acb6937eecde\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-9db8tm77q"
Nov 01 05:28:52.619397 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:52.619360    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8c6e7e1d-dfc3-49a1-a6e0-acb6937eecde-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-9db8tm77q\" (UID: \"8c6e7e1d-dfc3-49a1-a6e0-acb6937eecde\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-9db8tm77q"
Nov 01 05:28:52.619570 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:52.619503    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nz86\" (UniqueName: \"kubernetes.io/projected/8c6e7e1d-dfc3-49a1-a6e0-acb6937eecde-kube-api-access-5nz86\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-9db8tm77q\" (UID: \"8c6e7e1d-dfc3-49a1-a6e0-acb6937eecde\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-9db8tm77q"
Nov 01 05:28:52.620038 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:52.620015    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8c6e7e1d-dfc3-49a1-a6e0-acb6937eecde-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-9db8tm77q\" (UID: \"8c6e7e1d-dfc3-49a1-a6e0-acb6937eecde\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-9db8tm77q"
Nov 01 05:28:52.620112 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:52.620095    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c6e7e1d-dfc3-49a1-a6e0-acb6937eecde-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-9db8tm77q\" (UID: \"8c6e7e1d-dfc3-49a1-a6e0-acb6937eecde\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-9db8tm77q"
Nov 01 05:28:52.622409 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:52.622388    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8c6e7e1d-dfc3-49a1-a6e0-acb6937eecde-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-9db8tm77q\" (UID: \"8c6e7e1d-dfc3-49a1-a6e0-acb6937eecde\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-9db8tm77q"
Nov 01 05:28:52.680405 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:52.680380    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nz86\" (UniqueName: \"kubernetes.io/projected/8c6e7e1d-dfc3-49a1-a6e0-acb6937eecde-kube-api-access-5nz86\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-9db8tm77q\" (UID: \"8c6e7e1d-dfc3-49a1-a6e0-acb6937eecde\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-9db8tm77q"
Nov 01 05:28:52.695737 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:52.695715    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-9db8tm77q"
Nov 01 05:28:52.955532 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:52.955430    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7d1ec3/distinct-w0-7d1ec3-kserve-58cdfc8d58-n7xpt"
Nov 01 05:28:53.076420 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:53.076391    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-9db8tm77q"]
Nov 01 05:28:53.116814 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:28:53.116786    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c6e7e1d_dfc3_49a1_a6e0_acb6937eecde.slice/crio-af723480ece1fbd17e4523164267c0b9c4f7820d132ab4f80a68763c73dcabc2 WatchSource:0}: Error finding container af723480ece1fbd17e4523164267c0b9c4f7820d132ab4f80a68763c73dcabc2: Status 404 returned error can't find the container with id af723480ece1fbd17e4523164267c0b9c4f7820d132ab4f80a68763c73dcabc2
Nov 01 05:28:53.255105 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:53.255028    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:28:53.991828 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:53.991726    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-9db8tm77q" event={"ID":"8c6e7e1d-dfc3-49a1-a6e0-acb6937eecde","Type":"ContainerStarted","Data":"f24976884609d62e9ce0ae6bd4660364b3140fcb062085e4a84e0debb5af47b9"}
Nov 01 05:28:53.991828 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:53.991793    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-9db8tm77q" event={"ID":"8c6e7e1d-dfc3-49a1-a6e0-acb6937eecde","Type":"ContainerStarted","Data":"af723480ece1fbd17e4523164267c0b9c4f7820d132ab4f80a68763c73dcabc2"}
Nov 01 05:28:54.006494 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:54.006413    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-9db8tm77q"
Nov 01 05:28:54.226477 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:54.226300    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-9db8tm77q" podStartSLOduration=2.226277965 podStartE2EDuration="2.226277965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:28:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:28:54.22389109 +0000 UTC m=+1512.294687810" watchObservedRunningTime="2026-11-01 05:28:54.226277965 +0000 UTC m=+1512.297074687"
Nov 01 05:28:54.411676 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:54.397079    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:28:59.178590 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:28:59.178557    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:29:00.070758 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:29:00.070726    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:29:00.936801 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:29:00.936772    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-d989mcgpt"
Nov 01 05:29:02.211327 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:29:02.211298    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:29:02.386946 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:29:02.386912    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:29:03.064126 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:29:03.064097    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:29:04.006255 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:29:04.006214    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:29:05.016320 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:29:05.016294    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-9db8tm77q"
Nov 01 05:29:08.720266 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:29:08.720223    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:29:09.089916 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:29:09.089831    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:29:13.407465 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:29:13.407429    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:29:18.105315 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:29:18.105278    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:29:19.799831 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:29:19.799790    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:29:22.884680 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:29:22.884639    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:29:27.696868 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:29:27.696778    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:29:30.096884 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:29:30.096849    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:29:32.440837 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:29:32.440797    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:29:37.158190 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:29:37.158162    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:29:41.866120 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:29:41.866085    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:29:46.693679 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:29:46.693649    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:29:51.464693 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:29:51.460292    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:29:51.916641 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:29:51.916612    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws"]
Nov 01 05:29:51.920784 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:29:51.920767    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws"
Nov 01 05:29:51.961001 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:29:51.960969    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0eccaa64-f564-44d7-a611-9ff326b899b6-istio-envoy\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws\" (UID: \"0eccaa64-f564-44d7-a611-9ff326b899b6\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws"
Nov 01 05:29:51.961131 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:29:51.961021    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eccaa64-f564-44d7-a611-9ff326b899b6-credential-socket\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws\" (UID: \"0eccaa64-f564-44d7-a611-9ff326b899b6\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws"
Nov 01 05:29:51.961131 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:29:51.961074    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0eccaa64-f564-44d7-a611-9ff326b899b6-istio-podinfo\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws\" (UID: \"0eccaa64-f564-44d7-a611-9ff326b899b6\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws"
Nov 01 05:29:51.961208 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:29:51.961178    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cc9k6\" (UniqueName: \"kubernetes.io/projected/0eccaa64-f564-44d7-a611-9ff326b899b6-kube-api-access-cc9k6\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws\" (UID: \"0eccaa64-f564-44d7-a611-9ff326b899b6\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws"
Nov 01 05:29:51.961243 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:29:51.961214    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0eccaa64-f564-44d7-a611-9ff326b899b6-workload-certs\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws\" (UID: \"0eccaa64-f564-44d7-a611-9ff326b899b6\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws"
Nov 01 05:29:51.961290 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:29:51.961278    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0eccaa64-f564-44d7-a611-9ff326b899b6-istio-data\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws\" (UID: \"0eccaa64-f564-44d7-a611-9ff326b899b6\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws"
Nov 01 05:29:51.961340 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:29:51.961330    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0eccaa64-f564-44d7-a611-9ff326b899b6-istiod-ca-cert\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws\" (UID: \"0eccaa64-f564-44d7-a611-9ff326b899b6\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws"
Nov 01 05:29:51.961466 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:29:51.961441    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0eccaa64-f564-44d7-a611-9ff326b899b6-istio-token\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws\" (UID: \"0eccaa64-f564-44d7-a611-9ff326b899b6\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws"
Nov 01 05:29:51.961525 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:29:51.961512    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eccaa64-f564-44d7-a611-9ff326b899b6-workload-socket\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws\" (UID: \"0eccaa64-f564-44d7-a611-9ff326b899b6\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws"
Nov 01 05:29:52.021874 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:29:52.021834    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws"]
Nov 01 05:29:52.062737 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:29:52.062709    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eccaa64-f564-44d7-a611-9ff326b899b6-credential-socket\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws\" (UID: \"0eccaa64-f564-44d7-a611-9ff326b899b6\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws"
Nov 01 05:29:52.062929 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:29:52.062744    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0eccaa64-f564-44d7-a611-9ff326b899b6-istio-podinfo\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws\" (UID: \"0eccaa64-f564-44d7-a611-9ff326b899b6\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws"
Nov 01 05:29:52.062929 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:29:52.062808    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cc9k6\" (UniqueName: \"kubernetes.io/projected/0eccaa64-f564-44d7-a611-9ff326b899b6-kube-api-access-cc9k6\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws\" (UID: \"0eccaa64-f564-44d7-a611-9ff326b899b6\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws"
Nov 01 05:29:52.062929 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:29:52.062834    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0eccaa64-f564-44d7-a611-9ff326b899b6-workload-certs\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws\" (UID: \"0eccaa64-f564-44d7-a611-9ff326b899b6\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws"
Nov 01 05:29:52.062929 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:29:52.062896    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0eccaa64-f564-44d7-a611-9ff326b899b6-istio-data\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws\" (UID: \"0eccaa64-f564-44d7-a611-9ff326b899b6\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws"
Nov 01 05:29:52.063180 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:29:52.062975    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0eccaa64-f564-44d7-a611-9ff326b899b6-istiod-ca-cert\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws\" (UID: \"0eccaa64-f564-44d7-a611-9ff326b899b6\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws"
Nov 01 05:29:52.063180 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:29:52.063072    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0eccaa64-f564-44d7-a611-9ff326b899b6-istio-token\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws\" (UID: \"0eccaa64-f564-44d7-a611-9ff326b899b6\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws"
Nov 01 05:29:52.063180 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:29:52.063113    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eccaa64-f564-44d7-a611-9ff326b899b6-workload-socket\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws\" (UID: \"0eccaa64-f564-44d7-a611-9ff326b899b6\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws"
Nov 01 05:29:52.063180 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:29:52.063150    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0eccaa64-f564-44d7-a611-9ff326b899b6-istio-envoy\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws\" (UID: \"0eccaa64-f564-44d7-a611-9ff326b899b6\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws"
Nov 01 05:29:52.063180 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:29:52.063165    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eccaa64-f564-44d7-a611-9ff326b899b6-credential-socket\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws\" (UID: \"0eccaa64-f564-44d7-a611-9ff326b899b6\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws"
Nov 01 05:29:52.063446 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:29:52.063237    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0eccaa64-f564-44d7-a611-9ff326b899b6-workload-certs\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws\" (UID: \"0eccaa64-f564-44d7-a611-9ff326b899b6\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws"
Nov 01 05:29:52.063544 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:29:52.063527    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eccaa64-f564-44d7-a611-9ff326b899b6-workload-socket\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws\" (UID: \"0eccaa64-f564-44d7-a611-9ff326b899b6\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws"
Nov 01 05:29:52.063618 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:29:52.063550    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0eccaa64-f564-44d7-a611-9ff326b899b6-istio-data\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws\" (UID: \"0eccaa64-f564-44d7-a611-9ff326b899b6\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws"
Nov 01 05:29:52.063844 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:29:52.063815    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0eccaa64-f564-44d7-a611-9ff326b899b6-istiod-ca-cert\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws\" (UID: \"0eccaa64-f564-44d7-a611-9ff326b899b6\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws"
Nov 01 05:29:52.066146 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:29:52.066112    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0eccaa64-f564-44d7-a611-9ff326b899b6-istio-podinfo\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws\" (UID: \"0eccaa64-f564-44d7-a611-9ff326b899b6\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws"
Nov 01 05:29:52.067709 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:29:52.067687    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0eccaa64-f564-44d7-a611-9ff326b899b6-istio-envoy\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws\" (UID: \"0eccaa64-f564-44d7-a611-9ff326b899b6\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws"
Nov 01 05:29:52.123952 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:29:52.123931    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0eccaa64-f564-44d7-a611-9ff326b899b6-istio-token\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws\" (UID: \"0eccaa64-f564-44d7-a611-9ff326b899b6\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws"
Nov 01 05:29:52.124088 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:29:52.124066    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cc9k6\" (UniqueName: \"kubernetes.io/projected/0eccaa64-f564-44d7-a611-9ff326b899b6-kube-api-access-cc9k6\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws\" (UID: \"0eccaa64-f564-44d7-a611-9ff326b899b6\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws"
Nov 01 05:29:52.238784 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:29:52.238718    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws"
Nov 01 05:29:52.507496 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:29:52.507396    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:29:52.678320 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:29:52.678295    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws"]
Nov 01 05:29:52.680356 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:29:52.680330    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eccaa64_f564_44d7_a611_9ff326b899b6.slice/crio-4951cba639a60649727b4c79e78745eba1bf5fb4925a672936bdf779ffffb378 WatchSource:0}: Error finding container 4951cba639a60649727b4c79e78745eba1bf5fb4925a672936bdf779ffffb378: Status 404 returned error can't find the container with id 4951cba639a60649727b4c79e78745eba1bf5fb4925a672936bdf779ffffb378
Nov 01 05:29:52.682699 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:29:52.682668    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:29:52.682771 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:29:52.682732    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:29:52.682771 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:29:52.682762    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:29:53.372009 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:29:53.371971    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws" event={"ID":"0eccaa64-f564-44d7-a611-9ff326b899b6","Type":"ContainerStarted","Data":"ae218ab3dbdf626581a3f783a88ea00cef604ec42e1fa82ccea0faa4ff0b019d"}
Nov 01 05:29:53.372205 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:29:53.372021    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws" event={"ID":"0eccaa64-f564-44d7-a611-9ff326b899b6","Type":"ContainerStarted","Data":"4951cba639a60649727b4c79e78745eba1bf5fb4925a672936bdf779ffffb378"}
Nov 01 05:29:54.239112 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:29:54.239078    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws"
Nov 01 05:29:54.240226 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:29:54.240205    5689 patch_prober.go:29] interesting pod/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Nov 01 05:29:54.240286 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:29:54.240254    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws" podUID="0eccaa64-f564-44d7-a611-9ff326b899b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Nov 01 05:29:55.239246 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:29:55.239211    5689 patch_prober.go:29] interesting pod/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Nov 01 05:29:55.239651 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:29:55.239275    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws" podUID="0eccaa64-f564-44d7-a611-9ff326b899b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Nov 01 05:29:56.150746 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:29:56.150709    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:29:56.239887 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:29:56.239854    5689 patch_prober.go:29] interesting pod/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Nov 01 05:29:56.240274 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:29:56.239905    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws" podUID="0eccaa64-f564-44d7-a611-9ff326b899b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Nov 01 05:29:57.239737 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:29:57.239702    5689 patch_prober.go:29] interesting pod/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Nov 01 05:29:57.239916 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:29:57.239758    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws" podUID="0eccaa64-f564-44d7-a611-9ff326b899b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Nov 01 05:29:58.239077 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:29:58.239042    5689 patch_prober.go:29] interesting pod/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Nov 01 05:29:58.239244 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:29:58.239106    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws" podUID="0eccaa64-f564-44d7-a611-9ff326b899b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Nov 01 05:29:59.239621 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:29:59.239582    5689 patch_prober.go:29] interesting pod/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Nov 01 05:29:59.240012 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:29:59.239646    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws" podUID="0eccaa64-f564-44d7-a611-9ff326b899b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Nov 01 05:30:00.070590 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:30:00.070558    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:30:00.198349 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:00.198294    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws" podStartSLOduration=9.198280342 podStartE2EDuration="9.198280342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:29:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:29:53.502439434 +0000 UTC m=+1571.573236166" watchObservedRunningTime="2026-11-01 05:30:00.198280342 +0000 UTC m=+1578.269077061"
Nov 01 05:30:00.199789 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:00.199760    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891850-kql9d"]
Nov 01 05:30:00.204087 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:00.204065    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a12c35-29891850-wcvw2"]
Nov 01 05:30:00.204221 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:00.204197    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891850-kql9d"
Nov 01 05:30:00.207871 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:00.207854    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a12c35-29891850-wcvw2"
Nov 01 05:30:00.243685 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:00.243662    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws"
Nov 01 05:30:00.243989 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:00.243939    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws"
Nov 01 05:30:00.244636 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:00.244619    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws"
Nov 01 05:30:00.282110 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:00.282082    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzgxb\" (UniqueName: \"kubernetes.io/projected/880e86d4-6364-4e6d-9b59-103b1a8668aa-kube-api-access-qzgxb\") pod \"maas-api-key-cleanup-29891850-kql9d\" (UID: \"880e86d4-6364-4e6d-9b59-103b1a8668aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891850-kql9d"
Nov 01 05:30:00.282263 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:00.282143    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz7dl\" (UniqueName: \"kubernetes.io/projected/d217643f-7dbe-42bd-94c0-ae1b83d53cf8-kube-api-access-bz7dl\") pod \"maas-api-key-cleanup-e2e-extmod-a-a12c35-29891850-wcvw2\" (UID: \"d217643f-7dbe-42bd-94c0-ae1b83d53cf8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a12c35-29891850-wcvw2"
Nov 01 05:30:00.282366 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:00.282332    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d217643f-7dbe-42bd-94c0-ae1b83d53cf8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-a12c35-29891850-wcvw2\" (UID: \"d217643f-7dbe-42bd-94c0-ae1b83d53cf8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a12c35-29891850-wcvw2"
Nov 01 05:30:00.282531 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:00.282505    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/880e86d4-6364-4e6d-9b59-103b1a8668aa-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891850-kql9d\" (UID: \"880e86d4-6364-4e6d-9b59-103b1a8668aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891850-kql9d"
Nov 01 05:30:00.323614 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:00.322018    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891850-kql9d"]
Nov 01 05:30:00.324084 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:00.324047    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\""
Nov 01 05:30:00.325882 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:00.325855    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a12c35-29891850-wcvw2"]
Nov 01 05:30:00.384275 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:00.384240    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d217643f-7dbe-42bd-94c0-ae1b83d53cf8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-a12c35-29891850-wcvw2\" (UID: \"d217643f-7dbe-42bd-94c0-ae1b83d53cf8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a12c35-29891850-wcvw2"
Nov 01 05:30:00.384467 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:00.384350    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/880e86d4-6364-4e6d-9b59-103b1a8668aa-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891850-kql9d\" (UID: \"880e86d4-6364-4e6d-9b59-103b1a8668aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891850-kql9d"
Nov 01 05:30:00.384554 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:00.384466    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzgxb\" (UniqueName: \"kubernetes.io/projected/880e86d4-6364-4e6d-9b59-103b1a8668aa-kube-api-access-qzgxb\") pod \"maas-api-key-cleanup-29891850-kql9d\" (UID: \"880e86d4-6364-4e6d-9b59-103b1a8668aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891850-kql9d"
Nov 01 05:30:00.384554 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:00.384541    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz7dl\" (UniqueName: \"kubernetes.io/projected/d217643f-7dbe-42bd-94c0-ae1b83d53cf8-kube-api-access-bz7dl\") pod \"maas-api-key-cleanup-e2e-extmod-a-a12c35-29891850-wcvw2\" (UID: \"d217643f-7dbe-42bd-94c0-ae1b83d53cf8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a12c35-29891850-wcvw2"
Nov 01 05:30:00.385118 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:00.385087    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d217643f-7dbe-42bd-94c0-ae1b83d53cf8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-a12c35-29891850-wcvw2\" (UID: \"d217643f-7dbe-42bd-94c0-ae1b83d53cf8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a12c35-29891850-wcvw2"
Nov 01 05:30:00.385219 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:00.385198    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/880e86d4-6364-4e6d-9b59-103b1a8668aa-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891850-kql9d\" (UID: \"880e86d4-6364-4e6d-9b59-103b1a8668aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891850-kql9d"
Nov 01 05:30:00.445470 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:00.445418    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz7dl\" (UniqueName: \"kubernetes.io/projected/d217643f-7dbe-42bd-94c0-ae1b83d53cf8-kube-api-access-bz7dl\") pod \"maas-api-key-cleanup-e2e-extmod-a-a12c35-29891850-wcvw2\" (UID: \"d217643f-7dbe-42bd-94c0-ae1b83d53cf8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a12c35-29891850-wcvw2"
Nov 01 05:30:00.446267 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:00.446248    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzgxb\" (UniqueName: \"kubernetes.io/projected/880e86d4-6364-4e6d-9b59-103b1a8668aa-kube-api-access-qzgxb\") pod \"maas-api-key-cleanup-29891850-kql9d\" (UID: \"880e86d4-6364-4e6d-9b59-103b1a8668aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891850-kql9d"
Nov 01 05:30:00.520673 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:00.520642    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891850-kql9d"
Nov 01 05:30:00.525791 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:00.525765    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a12c35-29891850-wcvw2"
Nov 01 05:30:00.887892 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:00.887822    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:30:00.939473 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:00.938940    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891850-kql9d"]
Nov 01 05:30:00.972205 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:00.972073    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a12c35-29891850-wcvw2"]
Nov 01 05:30:00.976313 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:30:00.976282    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd217643f_7dbe_42bd_94c0_ae1b83d53cf8.slice/crio-b97bc57e48abea1aac5d71b6239a80a21e19d9de4d764b527756c94040d22797 WatchSource:0}: Error finding container b97bc57e48abea1aac5d71b6239a80a21e19d9de4d764b527756c94040d22797: Status 404 returned error can't find the container with id b97bc57e48abea1aac5d71b6239a80a21e19d9de4d764b527756c94040d22797
Nov 01 05:30:01.451034 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:01.450936    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a12c35-29891850-wcvw2" event={"ID":"d217643f-7dbe-42bd-94c0-ae1b83d53cf8","Type":"ContainerStarted","Data":"52fca2d8d65b41f7a78dfe7fb7103bc44433222882c00c564bf83fa3d5a23752"}
Nov 01 05:30:01.451034 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:01.451008    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a12c35-29891850-wcvw2" event={"ID":"d217643f-7dbe-42bd-94c0-ae1b83d53cf8","Type":"ContainerStarted","Data":"b97bc57e48abea1aac5d71b6239a80a21e19d9de4d764b527756c94040d22797"}
Nov 01 05:30:01.452788 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:01.452753    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891850-kql9d" event={"ID":"880e86d4-6364-4e6d-9b59-103b1a8668aa","Type":"ContainerStarted","Data":"0a2ea70004267d94b45a9d47462b77a98aff4ff04300082b42498df361f66f4c"}
Nov 01 05:30:01.452917 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:01.452806    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891850-kql9d" event={"ID":"880e86d4-6364-4e6d-9b59-103b1a8668aa","Type":"ContainerStarted","Data":"2182b04a58d2429cc0596b06db75877a65394b35a0c6c7a5a533ac00a2bd0ba2"}
Nov 01 05:30:01.581803 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:01.581757    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a12c35-29891850-wcvw2" podStartSLOduration=1.581742825 podStartE2EDuration="1.581742825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:30:01.579874206 +0000 UTC m=+1579.650670927" watchObservedRunningTime="2026-11-01 05:30:01.581742825 +0000 UTC m=+1579.652539557"
Nov 01 05:30:01.696053 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:01.695995    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891850-kql9d" podStartSLOduration=1.695981652 podStartE2EDuration="1.695981652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:30:01.693985431 +0000 UTC m=+1579.764782152" watchObservedRunningTime="2026-11-01 05:30:01.695981652 +0000 UTC m=+1579.766778369"
Nov 01 05:30:02.460954 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:02.460918    5689 generic.go:356] "Generic (PLEG): container finished" podID="d217643f-7dbe-42bd-94c0-ae1b83d53cf8" containerID="52fca2d8d65b41f7a78dfe7fb7103bc44433222882c00c564bf83fa3d5a23752" exitCode=60
Nov 01 05:30:02.461395 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:02.460997    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a12c35-29891850-wcvw2" event={"ID":"d217643f-7dbe-42bd-94c0-ae1b83d53cf8","Type":"ContainerDied","Data":"52fca2d8d65b41f7a78dfe7fb7103bc44433222882c00c564bf83fa3d5a23752"}
Nov 01 05:30:02.461395 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:02.461262    5689 scope.go:122] "RemoveContainer" containerID="52fca2d8d65b41f7a78dfe7fb7103bc44433222882c00c564bf83fa3d5a23752"
Nov 01 05:30:02.462769 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:02.462748    5689 generic.go:356] "Generic (PLEG): container finished" podID="880e86d4-6364-4e6d-9b59-103b1a8668aa" containerID="0a2ea70004267d94b45a9d47462b77a98aff4ff04300082b42498df361f66f4c" exitCode=60
Nov 01 05:30:02.462884 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:02.462832    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891850-kql9d" event={"ID":"880e86d4-6364-4e6d-9b59-103b1a8668aa","Type":"ContainerDied","Data":"0a2ea70004267d94b45a9d47462b77a98aff4ff04300082b42498df361f66f4c"}
Nov 01 05:30:02.463080 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:02.463062    5689 scope.go:122] "RemoveContainer" containerID="0a2ea70004267d94b45a9d47462b77a98aff4ff04300082b42498df361f66f4c"
Nov 01 05:30:02.562727 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:02.562689    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:30:03.470921 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:03.470847    5689 generic.go:356] "Generic (PLEG): container finished" podID="880e86d4-6364-4e6d-9b59-103b1a8668aa" containerID="03bdb964423ab5cc91fc05d3f33373c40113f3597a0d2625de941807ba8a1761" exitCode=60
Nov 01 05:30:03.471338 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:03.470969    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891850-kql9d" event={"ID":"880e86d4-6364-4e6d-9b59-103b1a8668aa","Type":"ContainerDied","Data":"03bdb964423ab5cc91fc05d3f33373c40113f3597a0d2625de941807ba8a1761"}
Nov 01 05:30:03.471338 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:03.471027    5689 scope.go:122] "RemoveContainer" containerID="0a2ea70004267d94b45a9d47462b77a98aff4ff04300082b42498df361f66f4c"
Nov 01 05:30:03.471338 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:03.471259    5689 scope.go:122] "RemoveContainer" containerID="03bdb964423ab5cc91fc05d3f33373c40113f3597a0d2625de941807ba8a1761"
Nov 01 05:30:03.471645 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:30:03.471613    5689 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891850-kql9d_odh-ai-gateway-infra(880e86d4-6364-4e6d-9b59-103b1a8668aa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891850-kql9d" podUID="880e86d4-6364-4e6d-9b59-103b1a8668aa"
Nov 01 05:30:03.472976 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:03.472957    5689 generic.go:356] "Generic (PLEG): container finished" podID="d217643f-7dbe-42bd-94c0-ae1b83d53cf8" containerID="0a12f24a22c9ec3df47c85e7b92eb68864ab3f2d338e85faa3474c4ac836e585" exitCode=60
Nov 01 05:30:03.473076 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:03.472982    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a12c35-29891850-wcvw2" event={"ID":"d217643f-7dbe-42bd-94c0-ae1b83d53cf8","Type":"ContainerDied","Data":"0a12f24a22c9ec3df47c85e7b92eb68864ab3f2d338e85faa3474c4ac836e585"}
Nov 01 05:30:03.473300 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:03.473283    5689 scope.go:122] "RemoveContainer" containerID="0a12f24a22c9ec3df47c85e7b92eb68864ab3f2d338e85faa3474c4ac836e585"
Nov 01 05:30:03.473578 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:30:03.473558    5689 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-a12c35-29891850-wcvw2_odh-ai-gateway-infra(d217643f-7dbe-42bd-94c0-ae1b83d53cf8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a12c35-29891850-wcvw2" podUID="d217643f-7dbe-42bd-94c0-ae1b83d53cf8"
Nov 01 05:30:03.483477 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:03.483433    5689 scope.go:122] "RemoveContainer" containerID="52fca2d8d65b41f7a78dfe7fb7103bc44433222882c00c564bf83fa3d5a23752"
Nov 01 05:30:04.480467 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:04.480424    5689 scope.go:122] "RemoveContainer" containerID="0a12f24a22c9ec3df47c85e7b92eb68864ab3f2d338e85faa3474c4ac836e585"
Nov 01 05:30:04.480885 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:30:04.480709    5689 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-a12c35-29891850-wcvw2_odh-ai-gateway-infra(d217643f-7dbe-42bd-94c0-ae1b83d53cf8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a12c35-29891850-wcvw2" podUID="d217643f-7dbe-42bd-94c0-ae1b83d53cf8"
Nov 01 05:30:04.482044 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:04.482022    5689 scope.go:122] "RemoveContainer" containerID="03bdb964423ab5cc91fc05d3f33373c40113f3597a0d2625de941807ba8a1761"
Nov 01 05:30:04.482306 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:30:04.482285    5689 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891850-kql9d_odh-ai-gateway-infra(880e86d4-6364-4e6d-9b59-103b1a8668aa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891850-kql9d" podUID="880e86d4-6364-4e6d-9b59-103b1a8668aa"
Nov 01 05:30:05.695335 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:05.695299    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:30:10.543031 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:10.542981    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:30:13.037310 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:13.037276    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-m5kl4"]
Nov 01 05:30:13.037833 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:13.037543    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-m5kl4" podUID="3ee14c22-76b6-4c41-bb64-c374e08328f1" containerName="istio-proxy" containerID="cri-o://585ad222babf72524a00368bfbcdd7aa655936a25f4c1006fa4f69f713fcdb3f" gracePeriod=30
Nov 01 05:30:13.086949 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:13.086914    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-sgdbb"]
Nov 01 05:30:13.087155 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:13.087113    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-sgdbb" podUID="3654852a-2923-46d4-a244-8a907545ed9c" containerName="istio-proxy" containerID="cri-o://583288b55e7c8e16beb20b204914acb34e85591bd8e203af8856448a4f40d826" gracePeriod=30
Nov 01 05:30:13.845169 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:13.845132    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90d244ee-g5tn4"]
Nov 01 05:30:13.849671 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:13.849648    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d244ee-g5tn4"
Nov 01 05:30:13.905149 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:13.905110    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 05:30:13.909219 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:13.909197    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90d244ee-g5tn4"]
Nov 01 05:30:13.976833 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:13.976802    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8374e5fe-7d4d-4a3c-824a-6447cac92b99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90d244ee-g5tn4\" (UID: \"8374e5fe-7d4d-4a3c-824a-6447cac92b99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d244ee-g5tn4"
Nov 01 05:30:13.976981 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:13.976863    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjmn9\" (UniqueName: \"kubernetes.io/projected/8374e5fe-7d4d-4a3c-824a-6447cac92b99-kube-api-access-tjmn9\") pod \"maas-api-revoke-sub-90d244ee-g5tn4\" (UID: \"8374e5fe-7d4d-4a3c-824a-6447cac92b99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d244ee-g5tn4"
Nov 01 05:30:13.977027 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:13.976968    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8374e5fe-7d4d-4a3c-824a-6447cac92b99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90d244ee-g5tn4\" (UID: \"8374e5fe-7d4d-4a3c-824a-6447cac92b99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d244ee-g5tn4"
Nov 01 05:30:14.078750 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:14.078721    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjmn9\" (UniqueName: \"kubernetes.io/projected/8374e5fe-7d4d-4a3c-824a-6447cac92b99-kube-api-access-tjmn9\") pod \"maas-api-revoke-sub-90d244ee-g5tn4\" (UID: \"8374e5fe-7d4d-4a3c-824a-6447cac92b99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d244ee-g5tn4"
Nov 01 05:30:14.079193 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:14.078776    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8374e5fe-7d4d-4a3c-824a-6447cac92b99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90d244ee-g5tn4\" (UID: \"8374e5fe-7d4d-4a3c-824a-6447cac92b99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d244ee-g5tn4"
Nov 01 05:30:14.079409 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:14.079385    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8374e5fe-7d4d-4a3c-824a-6447cac92b99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90d244ee-g5tn4\" (UID: \"8374e5fe-7d4d-4a3c-824a-6447cac92b99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d244ee-g5tn4"
Nov 01 05:30:14.079562 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:14.079541    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8374e5fe-7d4d-4a3c-824a-6447cac92b99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90d244ee-g5tn4\" (UID: \"8374e5fe-7d4d-4a3c-824a-6447cac92b99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d244ee-g5tn4"
Nov 01 05:30:14.081829 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:14.081805    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8374e5fe-7d4d-4a3c-824a-6447cac92b99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90d244ee-g5tn4\" (UID: \"8374e5fe-7d4d-4a3c-824a-6447cac92b99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d244ee-g5tn4"
Nov 01 05:30:14.139443 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:14.139387    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjmn9\" (UniqueName: \"kubernetes.io/projected/8374e5fe-7d4d-4a3c-824a-6447cac92b99-kube-api-access-tjmn9\") pod \"maas-api-revoke-sub-90d244ee-g5tn4\" (UID: \"8374e5fe-7d4d-4a3c-824a-6447cac92b99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d244ee-g5tn4"
Nov 01 05:30:14.163265 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:14.163244    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d244ee-g5tn4"
Nov 01 05:30:14.541199 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:14.541157    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90d244ee-g5tn4"]
Nov 01 05:30:14.549532 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:30:14.549499    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8374e5fe_7d4d_4a3c_824a_6447cac92b99.slice/crio-d612a22eb874bd47b05420feb4d9561831ae32adeed8af62f1743cfa9fdc3a90 WatchSource:0}: Error finding container d612a22eb874bd47b05420feb4d9561831ae32adeed8af62f1743cfa9fdc3a90: Status 404 returned error can't find the container with id d612a22eb874bd47b05420feb4d9561831ae32adeed8af62f1743cfa9fdc3a90
Nov 01 05:30:14.587705 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:14.587680    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d244ee-g5tn4" event={"ID":"8374e5fe-7d4d-4a3c-824a-6447cac92b99","Type":"ContainerStarted","Data":"d612a22eb874bd47b05420feb4d9561831ae32adeed8af62f1743cfa9fdc3a90"}
Nov 01 05:30:15.061304 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:15.061213    5689 scope.go:122] "RemoveContainer" containerID="0a12f24a22c9ec3df47c85e7b92eb68864ab3f2d338e85faa3474c4ac836e585"
Nov 01 05:30:15.061429 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:15.061380    5689 scope.go:122] "RemoveContainer" containerID="03bdb964423ab5cc91fc05d3f33373c40113f3597a0d2625de941807ba8a1761"
Nov 01 05:30:15.430956 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:15.430928    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:30:15.643720 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:15.643632    5689 generic.go:356] "Generic (PLEG): container finished" podID="8374e5fe-7d4d-4a3c-824a-6447cac92b99" containerID="412a661093710710b6e4b51c65196912cadaee5a187ebe7551256efb35cfe0ec" exitCode=0
Nov 01 05:30:15.644099 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:15.644070    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d244ee-g5tn4" event={"ID":"8374e5fe-7d4d-4a3c-824a-6447cac92b99","Type":"ContainerDied","Data":"412a661093710710b6e4b51c65196912cadaee5a187ebe7551256efb35cfe0ec"}
Nov 01 05:30:15.655578 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:15.655543    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a12c35-29891850-wcvw2" event={"ID":"d217643f-7dbe-42bd-94c0-ae1b83d53cf8","Type":"ContainerStarted","Data":"6ea6929f7ff94120e4cfac71c123f8e86490b9230ac9f4c4d90a3745c1bd9f9b"}
Nov 01 05:30:15.667515 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:15.667439    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891850-kql9d" event={"ID":"880e86d4-6364-4e6d-9b59-103b1a8668aa","Type":"ContainerStarted","Data":"f3a78cc3580225bcd3122751309b6e9287b3428850d94de07e9bbc3682a08bec"}
Nov 01 05:30:16.675529 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:16.675497    5689 generic.go:356] "Generic (PLEG): container finished" podID="d217643f-7dbe-42bd-94c0-ae1b83d53cf8" containerID="6ea6929f7ff94120e4cfac71c123f8e86490b9230ac9f4c4d90a3745c1bd9f9b" exitCode=60
Nov 01 05:30:16.675980 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:16.675570    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a12c35-29891850-wcvw2" event={"ID":"d217643f-7dbe-42bd-94c0-ae1b83d53cf8","Type":"ContainerDied","Data":"6ea6929f7ff94120e4cfac71c123f8e86490b9230ac9f4c4d90a3745c1bd9f9b"}
Nov 01 05:30:16.675980 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:16.675638    5689 scope.go:122] "RemoveContainer" containerID="0a12f24a22c9ec3df47c85e7b92eb68864ab3f2d338e85faa3474c4ac836e585"
Nov 01 05:30:16.676116 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:16.676003    5689 scope.go:122] "RemoveContainer" containerID="6ea6929f7ff94120e4cfac71c123f8e86490b9230ac9f4c4d90a3745c1bd9f9b"
Nov 01 05:30:16.676388 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:30:16.676349    5689 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-a12c35-29891850-wcvw2_odh-ai-gateway-infra(d217643f-7dbe-42bd-94c0-ae1b83d53cf8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a12c35-29891850-wcvw2" podUID="d217643f-7dbe-42bd-94c0-ae1b83d53cf8"
Nov 01 05:30:16.677780 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:16.677758    5689 generic.go:356] "Generic (PLEG): container finished" podID="880e86d4-6364-4e6d-9b59-103b1a8668aa" containerID="f3a78cc3580225bcd3122751309b6e9287b3428850d94de07e9bbc3682a08bec" exitCode=60
Nov 01 05:30:16.677882 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:16.677828    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891850-kql9d" event={"ID":"880e86d4-6364-4e6d-9b59-103b1a8668aa","Type":"ContainerDied","Data":"f3a78cc3580225bcd3122751309b6e9287b3428850d94de07e9bbc3682a08bec"}
Nov 01 05:30:16.678261 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:16.678234    5689 scope.go:122] "RemoveContainer" containerID="f3a78cc3580225bcd3122751309b6e9287b3428850d94de07e9bbc3682a08bec"
Nov 01 05:30:16.678513 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:30:16.678497    5689 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29891850-kql9d_odh-ai-gateway-infra(880e86d4-6364-4e6d-9b59-103b1a8668aa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891850-kql9d" podUID="880e86d4-6364-4e6d-9b59-103b1a8668aa"
Nov 01 05:30:16.687766 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:16.687744    5689 scope.go:122] "RemoveContainer" containerID="03bdb964423ab5cc91fc05d3f33373c40113f3597a0d2625de941807ba8a1761"
Nov 01 05:30:16.839784 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:16.839761    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d244ee-g5tn4"
Nov 01 05:30:16.884229 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:16.884201    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891850-kql9d"]
Nov 01 05:30:16.944822 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:16.944745    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8374e5fe-7d4d-4a3c-824a-6447cac92b99-kube-api-access-tjmn9\" (UniqueName: \"kubernetes.io/projected/8374e5fe-7d4d-4a3c-824a-6447cac92b99-kube-api-access-tjmn9\") pod \"8374e5fe-7d4d-4a3c-824a-6447cac92b99\" (UID: \"8374e5fe-7d4d-4a3c-824a-6447cac92b99\") "
Nov 01 05:30:16.944822 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:16.944790    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8374e5fe-7d4d-4a3c-824a-6447cac92b99-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8374e5fe-7d4d-4a3c-824a-6447cac92b99-maas-api-openshift-ca\") pod \"8374e5fe-7d4d-4a3c-824a-6447cac92b99\" (UID: \"8374e5fe-7d4d-4a3c-824a-6447cac92b99\") "
Nov 01 05:30:16.945045 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:16.945026    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8374e5fe-7d4d-4a3c-824a-6447cac92b99-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8374e5fe-7d4d-4a3c-824a-6447cac92b99-maas-api-xks-ca\") pod \"8374e5fe-7d4d-4a3c-824a-6447cac92b99\" (UID: \"8374e5fe-7d4d-4a3c-824a-6447cac92b99\") "
Nov 01 05:30:16.945169 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:16.945143    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8374e5fe-7d4d-4a3c-824a-6447cac92b99-maas-api-openshift-ca" pod "8374e5fe-7d4d-4a3c-824a-6447cac92b99" (UID: "8374e5fe-7d4d-4a3c-824a-6447cac92b99"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:30:16.945609 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:16.945529    5689 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8374e5fe-7d4d-4a3c-824a-6447cac92b99-maas-api-openshift-ca\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:30:16.947131 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:16.947105    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8374e5fe-7d4d-4a3c-824a-6447cac92b99-kube-api-access-tjmn9" pod "8374e5fe-7d4d-4a3c-824a-6447cac92b99" (UID: "8374e5fe-7d4d-4a3c-824a-6447cac92b99"). InnerVolumeSpecName "kube-api-access-tjmn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:30:16.947245 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:16.947155    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8374e5fe-7d4d-4a3c-824a-6447cac92b99-maas-api-xks-ca" pod "8374e5fe-7d4d-4a3c-824a-6447cac92b99" (UID: "8374e5fe-7d4d-4a3c-824a-6447cac92b99"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 05:30:16.998155 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:16.998130    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a12c35-29891850-wcvw2"]
Nov 01 05:30:17.046867 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:17.046836    5689 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8374e5fe-7d4d-4a3c-824a-6447cac92b99-maas-api-xks-ca\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:30:17.046867 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:17.046862    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjmn9\" (UniqueName: \"kubernetes.io/projected/8374e5fe-7d4d-4a3c-824a-6447cac92b99-kube-api-access-tjmn9\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:30:17.685769 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:17.685740    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d244ee-g5tn4"
Nov 01 05:30:17.686301 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:17.685775    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d244ee-g5tn4" event={"ID":"8374e5fe-7d4d-4a3c-824a-6447cac92b99","Type":"ContainerDied","Data":"d612a22eb874bd47b05420feb4d9561831ae32adeed8af62f1743cfa9fdc3a90"}
Nov 01 05:30:17.686301 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:17.685822    5689 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d612a22eb874bd47b05420feb4d9561831ae32adeed8af62f1743cfa9fdc3a90"
Nov 01 05:30:17.838104 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:17.838076    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891850-kql9d"
Nov 01 05:30:17.885023 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:17.885002    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a12c35-29891850-wcvw2"
Nov 01 05:30:17.958891 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:17.958815    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d217643f-7dbe-42bd-94c0-ae1b83d53cf8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d217643f-7dbe-42bd-94c0-ae1b83d53cf8-maas-api-service-ca\") pod \"d217643f-7dbe-42bd-94c0-ae1b83d53cf8\" (UID: \"d217643f-7dbe-42bd-94c0-ae1b83d53cf8\") "
Nov 01 05:30:17.958891 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:17.958854    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d217643f-7dbe-42bd-94c0-ae1b83d53cf8-kube-api-access-bz7dl\" (UniqueName: \"kubernetes.io/projected/d217643f-7dbe-42bd-94c0-ae1b83d53cf8-kube-api-access-bz7dl\") pod \"d217643f-7dbe-42bd-94c0-ae1b83d53cf8\" (UID: \"d217643f-7dbe-42bd-94c0-ae1b83d53cf8\") "
Nov 01 05:30:17.959074 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:17.958972    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/880e86d4-6364-4e6d-9b59-103b1a8668aa-kube-api-access-qzgxb\" (UniqueName: \"kubernetes.io/projected/880e86d4-6364-4e6d-9b59-103b1a8668aa-kube-api-access-qzgxb\") pod \"880e86d4-6364-4e6d-9b59-103b1a8668aa\" (UID: \"880e86d4-6364-4e6d-9b59-103b1a8668aa\") "
Nov 01 05:30:17.959074 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:17.959004    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/880e86d4-6364-4e6d-9b59-103b1a8668aa-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/880e86d4-6364-4e6d-9b59-103b1a8668aa-maas-api-service-ca\") pod \"880e86d4-6364-4e6d-9b59-103b1a8668aa\" (UID: \"880e86d4-6364-4e6d-9b59-103b1a8668aa\") "
Nov 01 05:30:17.959243 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:17.959219    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d217643f-7dbe-42bd-94c0-ae1b83d53cf8-maas-api-service-ca" pod "d217643f-7dbe-42bd-94c0-ae1b83d53cf8" (UID: "d217643f-7dbe-42bd-94c0-ae1b83d53cf8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:30:17.959467 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:17.959431    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/880e86d4-6364-4e6d-9b59-103b1a8668aa-maas-api-service-ca" pod "880e86d4-6364-4e6d-9b59-103b1a8668aa" (UID: "880e86d4-6364-4e6d-9b59-103b1a8668aa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:30:17.959708 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:17.959633    5689 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d217643f-7dbe-42bd-94c0-ae1b83d53cf8-maas-api-service-ca\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:30:17.959708 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:17.959651    5689 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/880e86d4-6364-4e6d-9b59-103b1a8668aa-maas-api-service-ca\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:30:17.961174 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:17.961156    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d217643f-7dbe-42bd-94c0-ae1b83d53cf8-kube-api-access-bz7dl" pod "d217643f-7dbe-42bd-94c0-ae1b83d53cf8" (UID: "d217643f-7dbe-42bd-94c0-ae1b83d53cf8"). InnerVolumeSpecName "kube-api-access-bz7dl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:30:17.961270 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:17.961184    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/880e86d4-6364-4e6d-9b59-103b1a8668aa-kube-api-access-qzgxb" pod "880e86d4-6364-4e6d-9b59-103b1a8668aa" (UID: "880e86d4-6364-4e6d-9b59-103b1a8668aa"). InnerVolumeSpecName "kube-api-access-qzgxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:30:18.061113 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:18.061085    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bz7dl\" (UniqueName: \"kubernetes.io/projected/d217643f-7dbe-42bd-94c0-ae1b83d53cf8-kube-api-access-bz7dl\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:30:18.061212 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:18.061121    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzgxb\" (UniqueName: \"kubernetes.io/projected/880e86d4-6364-4e6d-9b59-103b1a8668aa-kube-api-access-qzgxb\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:30:18.374840 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:18.374815    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-m5kl4"
Nov 01 05:30:18.378583 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:18.378564    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-sgdbb"
Nov 01 05:30:18.466661 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:18.466632    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3654852a-2923-46d4-a244-8a907545ed9c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3654852a-2923-46d4-a244-8a907545ed9c-workload-certs\") pod \"3654852a-2923-46d4-a244-8a907545ed9c\" (UID: \"3654852a-2923-46d4-a244-8a907545ed9c\") "
Nov 01 05:30:18.466822 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:18.466696    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ee14c22-76b6-4c41-bb64-c374e08328f1-istio-token\" (UniqueName: \"kubernetes.io/projected/3ee14c22-76b6-4c41-bb64-c374e08328f1-istio-token\") pod \"3ee14c22-76b6-4c41-bb64-c374e08328f1\" (UID: \"3ee14c22-76b6-4c41-bb64-c374e08328f1\") "
Nov 01 05:30:18.466822 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:18.466729    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3654852a-2923-46d4-a244-8a907545ed9c-kube-api-access-fb94c\" (UniqueName: \"kubernetes.io/projected/3654852a-2923-46d4-a244-8a907545ed9c-kube-api-access-fb94c\") pod \"3654852a-2923-46d4-a244-8a907545ed9c\" (UID: \"3654852a-2923-46d4-a244-8a907545ed9c\") "
Nov 01 05:30:18.466822 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:18.466763    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3654852a-2923-46d4-a244-8a907545ed9c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3654852a-2923-46d4-a244-8a907545ed9c-istio-data\") pod \"3654852a-2923-46d4-a244-8a907545ed9c\" (UID: \"3654852a-2923-46d4-a244-8a907545ed9c\") "
Nov 01 05:30:18.466822 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:18.466791    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3ee14c22-76b6-4c41-bb64-c374e08328f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ee14c22-76b6-4c41-bb64-c374e08328f1-istio-podinfo\") pod \"3ee14c22-76b6-4c41-bb64-c374e08328f1\" (UID: \"3ee14c22-76b6-4c41-bb64-c374e08328f1\") "
Nov 01 05:30:18.467040 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:18.466848    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ee14c22-76b6-4c41-bb64-c374e08328f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ee14c22-76b6-4c41-bb64-c374e08328f1-istio-data\") pod \"3ee14c22-76b6-4c41-bb64-c374e08328f1\" (UID: \"3ee14c22-76b6-4c41-bb64-c374e08328f1\") "
Nov 01 05:30:18.467040 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:18.466928    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3654852a-2923-46d4-a244-8a907545ed9c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3654852a-2923-46d4-a244-8a907545ed9c-workload-socket\") pod \"3654852a-2923-46d4-a244-8a907545ed9c\" (UID: \"3654852a-2923-46d4-a244-8a907545ed9c\") "
Nov 01 05:30:18.467040 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:18.466930    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3654852a-2923-46d4-a244-8a907545ed9c-workload-certs" pod "3654852a-2923-46d4-a244-8a907545ed9c" (UID: "3654852a-2923-46d4-a244-8a907545ed9c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:30:18.467040 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:18.467024    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ee14c22-76b6-4c41-bb64-c374e08328f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ee14c22-76b6-4c41-bb64-c374e08328f1-credential-socket\") pod \"3ee14c22-76b6-4c41-bb64-c374e08328f1\" (UID: \"3ee14c22-76b6-4c41-bb64-c374e08328f1\") "
Nov 01 05:30:18.467257 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:18.467078    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ee14c22-76b6-4c41-bb64-c374e08328f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ee14c22-76b6-4c41-bb64-c374e08328f1-istiod-ca-cert\") pod \"3ee14c22-76b6-4c41-bb64-c374e08328f1\" (UID: \"3ee14c22-76b6-4c41-bb64-c374e08328f1\") "
Nov 01 05:30:18.467257 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:18.467097    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3654852a-2923-46d4-a244-8a907545ed9c-istio-data" pod "3654852a-2923-46d4-a244-8a907545ed9c" (UID: "3654852a-2923-46d4-a244-8a907545ed9c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:30:18.467257 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:18.467113    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3654852a-2923-46d4-a244-8a907545ed9c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3654852a-2923-46d4-a244-8a907545ed9c-istiod-ca-cert\") pod \"3654852a-2923-46d4-a244-8a907545ed9c\" (UID: \"3654852a-2923-46d4-a244-8a907545ed9c\") "
Nov 01 05:30:18.467257 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:18.467133    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3654852a-2923-46d4-a244-8a907545ed9c-workload-socket" pod "3654852a-2923-46d4-a244-8a907545ed9c" (UID: "3654852a-2923-46d4-a244-8a907545ed9c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:30:18.467257 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:18.467153    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ee14c22-76b6-4c41-bb64-c374e08328f1-kube-api-access-t4j9l\" (UniqueName: \"kubernetes.io/projected/3ee14c22-76b6-4c41-bb64-c374e08328f1-kube-api-access-t4j9l\") pod \"3ee14c22-76b6-4c41-bb64-c374e08328f1\" (UID: \"3ee14c22-76b6-4c41-bb64-c374e08328f1\") "
Nov 01 05:30:18.467257 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:18.467218    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3654852a-2923-46d4-a244-8a907545ed9c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3654852a-2923-46d4-a244-8a907545ed9c-istio-envoy\") pod \"3654852a-2923-46d4-a244-8a907545ed9c\" (UID: \"3654852a-2923-46d4-a244-8a907545ed9c\") "
Nov 01 05:30:18.467689 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:18.467284    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ee14c22-76b6-4c41-bb64-c374e08328f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ee14c22-76b6-4c41-bb64-c374e08328f1-workload-certs\") pod \"3ee14c22-76b6-4c41-bb64-c374e08328f1\" (UID: \"3ee14c22-76b6-4c41-bb64-c374e08328f1\") "
Nov 01 05:30:18.467689 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:18.467285    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ee14c22-76b6-4c41-bb64-c374e08328f1-istio-data" pod "3ee14c22-76b6-4c41-bb64-c374e08328f1" (UID: "3ee14c22-76b6-4c41-bb64-c374e08328f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:30:18.467689 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:18.467300    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ee14c22-76b6-4c41-bb64-c374e08328f1-credential-socket" pod "3ee14c22-76b6-4c41-bb64-c374e08328f1" (UID: "3ee14c22-76b6-4c41-bb64-c374e08328f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:30:18.467689 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:18.467346    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3654852a-2923-46d4-a244-8a907545ed9c-istio-token\" (UniqueName: \"kubernetes.io/projected/3654852a-2923-46d4-a244-8a907545ed9c-istio-token\") pod \"3654852a-2923-46d4-a244-8a907545ed9c\" (UID: \"3654852a-2923-46d4-a244-8a907545ed9c\") "
Nov 01 05:30:18.467689 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:18.467392    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3654852a-2923-46d4-a244-8a907545ed9c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3654852a-2923-46d4-a244-8a907545ed9c-istio-podinfo\") pod \"3654852a-2923-46d4-a244-8a907545ed9c\" (UID: \"3654852a-2923-46d4-a244-8a907545ed9c\") "
Nov 01 05:30:18.467689 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:18.467469    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ee14c22-76b6-4c41-bb64-c374e08328f1-istiod-ca-cert" pod "3ee14c22-76b6-4c41-bb64-c374e08328f1" (UID: "3ee14c22-76b6-4c41-bb64-c374e08328f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:30:18.467689 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:18.467482    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3654852a-2923-46d4-a244-8a907545ed9c-istiod-ca-cert" pod "3654852a-2923-46d4-a244-8a907545ed9c" (UID: "3654852a-2923-46d4-a244-8a907545ed9c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:30:18.467689 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:18.467486    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3654852a-2923-46d4-a244-8a907545ed9c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3654852a-2923-46d4-a244-8a907545ed9c-credential-socket\") pod \"3654852a-2923-46d4-a244-8a907545ed9c\" (UID: \"3654852a-2923-46d4-a244-8a907545ed9c\") "
Nov 01 05:30:18.467689 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:18.467519    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ee14c22-76b6-4c41-bb64-c374e08328f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ee14c22-76b6-4c41-bb64-c374e08328f1-workload-socket\") pod \"3ee14c22-76b6-4c41-bb64-c374e08328f1\" (UID: \"3ee14c22-76b6-4c41-bb64-c374e08328f1\") "
Nov 01 05:30:18.467689 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:18.467553    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ee14c22-76b6-4c41-bb64-c374e08328f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ee14c22-76b6-4c41-bb64-c374e08328f1-istio-envoy\") pod \"3ee14c22-76b6-4c41-bb64-c374e08328f1\" (UID: \"3ee14c22-76b6-4c41-bb64-c374e08328f1\") "
Nov 01 05:30:18.468354 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:18.467707    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ee14c22-76b6-4c41-bb64-c374e08328f1-workload-certs" pod "3ee14c22-76b6-4c41-bb64-c374e08328f1" (UID: "3ee14c22-76b6-4c41-bb64-c374e08328f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:30:18.468354 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:18.467837    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3654852a-2923-46d4-a244-8a907545ed9c-credential-socket" pod "3654852a-2923-46d4-a244-8a907545ed9c" (UID: "3654852a-2923-46d4-a244-8a907545ed9c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:30:18.468354 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:18.467887    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ee14c22-76b6-4c41-bb64-c374e08328f1-workload-socket" pod "3ee14c22-76b6-4c41-bb64-c374e08328f1" (UID: "3ee14c22-76b6-4c41-bb64-c374e08328f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:30:18.468354 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:18.468206    5689 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ee14c22-76b6-4c41-bb64-c374e08328f1-workload-socket\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:30:18.468354 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:18.468231    5689 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3654852a-2923-46d4-a244-8a907545ed9c-workload-certs\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:30:18.468354 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:18.468247    5689 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3654852a-2923-46d4-a244-8a907545ed9c-istio-data\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:30:18.468354 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:18.468261    5689 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ee14c22-76b6-4c41-bb64-c374e08328f1-istio-data\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:30:18.468354 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:18.468276    5689 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3654852a-2923-46d4-a244-8a907545ed9c-workload-socket\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:30:18.468354 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:18.468290    5689 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ee14c22-76b6-4c41-bb64-c374e08328f1-credential-socket\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:30:18.468354 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:18.468304    5689 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ee14c22-76b6-4c41-bb64-c374e08328f1-istiod-ca-cert\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:30:18.468354 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:18.468319    5689 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3654852a-2923-46d4-a244-8a907545ed9c-istiod-ca-cert\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:30:18.468354 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:18.468337    5689 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ee14c22-76b6-4c41-bb64-c374e08328f1-workload-certs\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:30:18.468354 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:18.468353    5689 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3654852a-2923-46d4-a244-8a907545ed9c-credential-socket\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:30:18.470738 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:18.470698    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ee14c22-76b6-4c41-bb64-c374e08328f1-istio-envoy" pod "3ee14c22-76b6-4c41-bb64-c374e08328f1" (UID: "3ee14c22-76b6-4c41-bb64-c374e08328f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:30:18.470738 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:18.470698    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3654852a-2923-46d4-a244-8a907545ed9c-kube-api-access-fb94c" pod "3654852a-2923-46d4-a244-8a907545ed9c" (UID: "3654852a-2923-46d4-a244-8a907545ed9c"). InnerVolumeSpecName "kube-api-access-fb94c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:30:18.470919 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:18.470784    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3654852a-2923-46d4-a244-8a907545ed9c-istio-token" pod "3654852a-2923-46d4-a244-8a907545ed9c" (UID: "3654852a-2923-46d4-a244-8a907545ed9c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:30:18.470919 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:18.470797    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3654852a-2923-46d4-a244-8a907545ed9c-istio-envoy" pod "3654852a-2923-46d4-a244-8a907545ed9c" (UID: "3654852a-2923-46d4-a244-8a907545ed9c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:30:18.470919 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:18.470850    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ee14c22-76b6-4c41-bb64-c374e08328f1-istio-token" pod "3ee14c22-76b6-4c41-bb64-c374e08328f1" (UID: "3ee14c22-76b6-4c41-bb64-c374e08328f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:30:18.471060 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:18.471022    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3ee14c22-76b6-4c41-bb64-c374e08328f1-istio-podinfo" pod "3ee14c22-76b6-4c41-bb64-c374e08328f1" (UID: "3ee14c22-76b6-4c41-bb64-c374e08328f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 05:30:18.471420 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:18.471398    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ee14c22-76b6-4c41-bb64-c374e08328f1-kube-api-access-t4j9l" pod "3ee14c22-76b6-4c41-bb64-c374e08328f1" (UID: "3ee14c22-76b6-4c41-bb64-c374e08328f1"). InnerVolumeSpecName "kube-api-access-t4j9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:30:18.471642 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:18.471626    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3654852a-2923-46d4-a244-8a907545ed9c-istio-podinfo" pod "3654852a-2923-46d4-a244-8a907545ed9c" (UID: "3654852a-2923-46d4-a244-8a907545ed9c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 05:30:18.569798 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:18.569724    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4j9l\" (UniqueName: \"kubernetes.io/projected/3ee14c22-76b6-4c41-bb64-c374e08328f1-kube-api-access-t4j9l\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:30:18.569798 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:18.569753    5689 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3654852a-2923-46d4-a244-8a907545ed9c-istio-envoy\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:30:18.569798 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:18.569762    5689 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3654852a-2923-46d4-a244-8a907545ed9c-istio-token\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:30:18.569798 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:18.569771    5689 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3654852a-2923-46d4-a244-8a907545ed9c-istio-podinfo\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:30:18.569798 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:18.569780    5689 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ee14c22-76b6-4c41-bb64-c374e08328f1-istio-envoy\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:30:18.569798 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:18.569788    5689 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ee14c22-76b6-4c41-bb64-c374e08328f1-istio-token\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:30:18.569798 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:18.569796    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fb94c\" (UniqueName: \"kubernetes.io/projected/3654852a-2923-46d4-a244-8a907545ed9c-kube-api-access-fb94c\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:30:18.570111 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:18.569805    5689 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ee14c22-76b6-4c41-bb64-c374e08328f1-istio-podinfo\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:30:18.696461 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:18.696428    5689 generic.go:356] "Generic (PLEG): container finished" podID="3654852a-2923-46d4-a244-8a907545ed9c" containerID="583288b55e7c8e16beb20b204914acb34e85591bd8e203af8856448a4f40d826" exitCode=0
Nov 01 05:30:18.696870 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:18.696510    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-sgdbb"
Nov 01 05:30:18.696870 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:18.696515    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-sgdbb" event={"ID":"3654852a-2923-46d4-a244-8a907545ed9c","Type":"ContainerDied","Data":"583288b55e7c8e16beb20b204914acb34e85591bd8e203af8856448a4f40d826"}
Nov 01 05:30:18.696870 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:18.696573    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-sgdbb" event={"ID":"3654852a-2923-46d4-a244-8a907545ed9c","Type":"ContainerDied","Data":"326d0f60ab833bd9980e10c231569a040244f7a0671dab0615957bc3b75b0b93"}
Nov 01 05:30:18.696870 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:18.696603    5689 scope.go:122] "RemoveContainer" containerID="583288b55e7c8e16beb20b204914acb34e85591bd8e203af8856448a4f40d826"
Nov 01 05:30:18.698054 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:18.698033    5689 generic.go:356] "Generic (PLEG): container finished" podID="3ee14c22-76b6-4c41-bb64-c374e08328f1" containerID="585ad222babf72524a00368bfbcdd7aa655936a25f4c1006fa4f69f713fcdb3f" exitCode=0
Nov 01 05:30:18.698142 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:18.698061    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-m5kl4" event={"ID":"3ee14c22-76b6-4c41-bb64-c374e08328f1","Type":"ContainerDied","Data":"585ad222babf72524a00368bfbcdd7aa655936a25f4c1006fa4f69f713fcdb3f"}
Nov 01 05:30:18.698142 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:18.698090    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-m5kl4"
Nov 01 05:30:18.698142 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:18.698113    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-m5kl4" event={"ID":"3ee14c22-76b6-4c41-bb64-c374e08328f1","Type":"ContainerDied","Data":"4104beb1d7f6dfb7d8d3d5d13ae357ef30a21df188077e86e0623104f25cab34"}
Nov 01 05:30:18.699865 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:18.699845    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a12c35-29891850-wcvw2"
Nov 01 05:30:18.700489 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:18.700271    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a12c35-29891850-wcvw2" event={"ID":"d217643f-7dbe-42bd-94c0-ae1b83d53cf8","Type":"ContainerDied","Data":"b97bc57e48abea1aac5d71b6239a80a21e19d9de4d764b527756c94040d22797"}
Nov 01 05:30:18.701295 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:18.701279    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891850-kql9d"
Nov 01 05:30:18.701406 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:18.701280    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891850-kql9d" event={"ID":"880e86d4-6364-4e6d-9b59-103b1a8668aa","Type":"ContainerDied","Data":"2182b04a58d2429cc0596b06db75877a65394b35a0c6c7a5a533ac00a2bd0ba2"}
Nov 01 05:30:18.708843 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:18.708823    5689 scope.go:122] "RemoveContainer" containerID="583288b55e7c8e16beb20b204914acb34e85591bd8e203af8856448a4f40d826"
Nov 01 05:30:18.709137 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:30:18.709094    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"583288b55e7c8e16beb20b204914acb34e85591bd8e203af8856448a4f40d826\": container with ID starting with 583288b55e7c8e16beb20b204914acb34e85591bd8e203af8856448a4f40d826 not found: ID does not exist" containerID="583288b55e7c8e16beb20b204914acb34e85591bd8e203af8856448a4f40d826"
Nov 01 05:30:18.709137 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:18.709122    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"583288b55e7c8e16beb20b204914acb34e85591bd8e203af8856448a4f40d826"} err="failed to get container status \"583288b55e7c8e16beb20b204914acb34e85591bd8e203af8856448a4f40d826\": rpc error: code = NotFound desc = could not find container \"583288b55e7c8e16beb20b204914acb34e85591bd8e203af8856448a4f40d826\": container with ID starting with 583288b55e7c8e16beb20b204914acb34e85591bd8e203af8856448a4f40d826 not found: ID does not exist"
Nov 01 05:30:18.709220 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:18.709146    5689 scope.go:122] "RemoveContainer" containerID="585ad222babf72524a00368bfbcdd7aa655936a25f4c1006fa4f69f713fcdb3f"
Nov 01 05:30:18.718443 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:18.718427    5689 scope.go:122] "RemoveContainer" containerID="585ad222babf72524a00368bfbcdd7aa655936a25f4c1006fa4f69f713fcdb3f"
Nov 01 05:30:18.718746 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:30:18.718727    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"585ad222babf72524a00368bfbcdd7aa655936a25f4c1006fa4f69f713fcdb3f\": container with ID starting with 585ad222babf72524a00368bfbcdd7aa655936a25f4c1006fa4f69f713fcdb3f not found: ID does not exist" containerID="585ad222babf72524a00368bfbcdd7aa655936a25f4c1006fa4f69f713fcdb3f"
Nov 01 05:30:18.718810 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:18.718752    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"585ad222babf72524a00368bfbcdd7aa655936a25f4c1006fa4f69f713fcdb3f"} err="failed to get container status \"585ad222babf72524a00368bfbcdd7aa655936a25f4c1006fa4f69f713fcdb3f\": rpc error: code = NotFound desc = could not find container \"585ad222babf72524a00368bfbcdd7aa655936a25f4c1006fa4f69f713fcdb3f\": container with ID starting with 585ad222babf72524a00368bfbcdd7aa655936a25f4c1006fa4f69f713fcdb3f not found: ID does not exist"
Nov 01 05:30:18.718810 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:18.718769    5689 scope.go:122] "RemoveContainer" containerID="6ea6929f7ff94120e4cfac71c123f8e86490b9230ac9f4c4d90a3745c1bd9f9b"
Nov 01 05:30:18.734645 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:18.734621    5689 scope.go:122] "RemoveContainer" containerID="f3a78cc3580225bcd3122751309b6e9287b3428850d94de07e9bbc3682a08bec"
Nov 01 05:30:18.864481 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:18.864396    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-m5kl4"]
Nov 01 05:30:18.868520 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:18.868496    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-m5kl4"]
Nov 01 05:30:18.977462 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:18.977430    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-sgdbb"]
Nov 01 05:30:18.986020 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:18.985993    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-sgdbb"]
Nov 01 05:30:19.093156 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:19.093127    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a12c35-29891850-wcvw2"]
Nov 01 05:30:19.101074 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:19.101045    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a12c35-29891850-wcvw2"]
Nov 01 05:30:19.209623 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:19.209583    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891850-kql9d"]
Nov 01 05:30:19.213269 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:19.213244    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891850-kql9d"]
Nov 01 05:30:19.267095 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:19.267062    5689 patch_prober.go:29] interesting pod/e2e-worker-w4-a48099-openshift-default-577864759-m5kl4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 05:30:19.267239 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:19.267123    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-m5kl4" podUID="3ee14c22-76b6-4c41-bb64-c374e08328f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 05:30:20.068655 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:20.068624    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3654852a-2923-46d4-a244-8a907545ed9c" path="/var/lib/kubelet/pods/3654852a-2923-46d4-a244-8a907545ed9c/volumes"
Nov 01 05:30:20.069101 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:20.069086    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ee14c22-76b6-4c41-bb64-c374e08328f1" path="/var/lib/kubelet/pods/3ee14c22-76b6-4c41-bb64-c374e08328f1/volumes"
Nov 01 05:30:20.069448 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:20.069436    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="880e86d4-6364-4e6d-9b59-103b1a8668aa" path="/var/lib/kubelet/pods/880e86d4-6364-4e6d-9b59-103b1a8668aa/volumes"
Nov 01 05:30:20.070134 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:20.070119    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d217643f-7dbe-42bd-94c0-ae1b83d53cf8" path="/var/lib/kubelet/pods/d217643f-7dbe-42bd-94c0-ae1b83d53cf8/volumes"
Nov 01 05:30:20.245699 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:20.245666    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:30:25.366479 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:25.365489    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:30:28.206867 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:28.206834    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:30:30.070289 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:30:30.070254    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:30:30.227476 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:30.226843    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:30:32.484883 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:32.484842    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:30:33.225010 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:33.224948    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:30:33.434934 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:33.434904    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:30:34.783258 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:34.783220    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:30:35.199937 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:35.199864    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:30:39.824943 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:39.824903    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:30:40.012431 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:40.012396    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:30:44.132934 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:44.132895    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:30:44.865050 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:44.853416    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:30:49.639488 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:49.629727    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:30:49.707392 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:49.707356    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5lvz97"]
Nov 01 05:30:51.921583 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:51.921500    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl"]
Nov 01 05:30:51.922040 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:51.922026    5689 cpu_manager.go:414] "containerMap: removing container" podUID="3654852a-2923-46d4-a244-8a907545ed9c" containerName="istio-proxy"
Nov 01 05:30:51.922040 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:51.922041    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3654852a-2923-46d4-a244-8a907545ed9c" containerName="istio-proxy"
Nov 01 05:30:51.922122 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:51.922050    5689 cpu_manager.go:414] "containerMap: removing container" podUID="d217643f-7dbe-42bd-94c0-ae1b83d53cf8" containerName="cleanup"
Nov 01 05:30:51.922122 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:51.922055    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d217643f-7dbe-42bd-94c0-ae1b83d53cf8" containerName="cleanup"
Nov 01 05:30:51.922122 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:51.922074    5689 cpu_manager.go:414] "containerMap: removing container" podUID="d217643f-7dbe-42bd-94c0-ae1b83d53cf8" containerName="cleanup"
Nov 01 05:30:51.922122 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:51.922078    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d217643f-7dbe-42bd-94c0-ae1b83d53cf8" containerName="cleanup"
Nov 01 05:30:51.922122 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:51.922085    5689 cpu_manager.go:414] "containerMap: removing container" podUID="d217643f-7dbe-42bd-94c0-ae1b83d53cf8" containerName="cleanup"
Nov 01 05:30:51.922122 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:51.922089    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d217643f-7dbe-42bd-94c0-ae1b83d53cf8" containerName="cleanup"
Nov 01 05:30:51.922122 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:51.922097    5689 cpu_manager.go:414] "containerMap: removing container" podUID="3ee14c22-76b6-4c41-bb64-c374e08328f1" containerName="istio-proxy"
Nov 01 05:30:51.922122 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:51.922101    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ee14c22-76b6-4c41-bb64-c374e08328f1" containerName="istio-proxy"
Nov 01 05:30:51.922122 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:51.922118    5689 cpu_manager.go:414] "containerMap: removing container" podUID="880e86d4-6364-4e6d-9b59-103b1a8668aa" containerName="cleanup"
Nov 01 05:30:51.922122 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:51.922122    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="880e86d4-6364-4e6d-9b59-103b1a8668aa" containerName="cleanup"
Nov 01 05:30:51.922417 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:51.922132    5689 cpu_manager.go:414] "containerMap: removing container" podUID="880e86d4-6364-4e6d-9b59-103b1a8668aa" containerName="cleanup"
Nov 01 05:30:51.922417 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:51.922137    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="880e86d4-6364-4e6d-9b59-103b1a8668aa" containerName="cleanup"
Nov 01 05:30:51.922417 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:51.922142    5689 cpu_manager.go:414] "containerMap: removing container" podUID="8374e5fe-7d4d-4a3c-824a-6447cac92b99" containerName="revoke-keys"
Nov 01 05:30:51.922417 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:51.922147    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8374e5fe-7d4d-4a3c-824a-6447cac92b99" containerName="revoke-keys"
Nov 01 05:30:51.922417 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:51.922227    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="8374e5fe-7d4d-4a3c-824a-6447cac92b99" containerName="revoke-keys"
Nov 01 05:30:51.922417 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:51.922236    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="880e86d4-6364-4e6d-9b59-103b1a8668aa" containerName="cleanup"
Nov 01 05:30:51.922417 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:51.922244    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="3654852a-2923-46d4-a244-8a907545ed9c" containerName="istio-proxy"
Nov 01 05:30:51.922417 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:51.922249    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="d217643f-7dbe-42bd-94c0-ae1b83d53cf8" containerName="cleanup"
Nov 01 05:30:51.922417 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:51.922257    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="d217643f-7dbe-42bd-94c0-ae1b83d53cf8" containerName="cleanup"
Nov 01 05:30:51.922417 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:51.922261    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="d217643f-7dbe-42bd-94c0-ae1b83d53cf8" containerName="cleanup"
Nov 01 05:30:51.922417 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:51.922267    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ee14c22-76b6-4c41-bb64-c374e08328f1" containerName="istio-proxy"
Nov 01 05:30:51.922417 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:51.922276    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="880e86d4-6364-4e6d-9b59-103b1a8668aa" containerName="cleanup"
Nov 01 05:30:51.927091 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:51.927075    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl"
Nov 01 05:30:52.068775 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:52.068743    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c7f07f0-82ce-4e5b-97c5-946cc901566d-credential-socket\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl\" (UID: \"4c7f07f0-82ce-4e5b-97c5-946cc901566d\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl"
Nov 01 05:30:52.068936 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:52.068793    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c7f07f0-82ce-4e5b-97c5-946cc901566d-istio-podinfo\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl\" (UID: \"4c7f07f0-82ce-4e5b-97c5-946cc901566d\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl"
Nov 01 05:30:52.068936 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:52.068865    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c7f07f0-82ce-4e5b-97c5-946cc901566d-istio-data\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl\" (UID: \"4c7f07f0-82ce-4e5b-97c5-946cc901566d\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl"
Nov 01 05:30:52.068936 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:52.068918    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wpdk\" (UniqueName: \"kubernetes.io/projected/4c7f07f0-82ce-4e5b-97c5-946cc901566d-kube-api-access-8wpdk\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl\" (UID: \"4c7f07f0-82ce-4e5b-97c5-946cc901566d\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl"
Nov 01 05:30:52.069071 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:52.068960    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c7f07f0-82ce-4e5b-97c5-946cc901566d-workload-socket\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl\" (UID: \"4c7f07f0-82ce-4e5b-97c5-946cc901566d\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl"
Nov 01 05:30:52.069071 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:52.068985    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c7f07f0-82ce-4e5b-97c5-946cc901566d-istiod-ca-cert\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl\" (UID: \"4c7f07f0-82ce-4e5b-97c5-946cc901566d\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl"
Nov 01 05:30:52.069134 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:52.069093    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c7f07f0-82ce-4e5b-97c5-946cc901566d-workload-certs\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl\" (UID: \"4c7f07f0-82ce-4e5b-97c5-946cc901566d\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl"
Nov 01 05:30:52.069167 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:52.069133    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c7f07f0-82ce-4e5b-97c5-946cc901566d-istio-token\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl\" (UID: \"4c7f07f0-82ce-4e5b-97c5-946cc901566d\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl"
Nov 01 05:30:52.069245 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:52.069230    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c7f07f0-82ce-4e5b-97c5-946cc901566d-istio-envoy\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl\" (UID: \"4c7f07f0-82ce-4e5b-97c5-946cc901566d\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl"
Nov 01 05:30:52.132846 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:52.132814    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl"]
Nov 01 05:30:52.171010 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:52.170979    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wpdk\" (UniqueName: \"kubernetes.io/projected/4c7f07f0-82ce-4e5b-97c5-946cc901566d-kube-api-access-8wpdk\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl\" (UID: \"4c7f07f0-82ce-4e5b-97c5-946cc901566d\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl"
Nov 01 05:30:52.171187 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:52.171055    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c7f07f0-82ce-4e5b-97c5-946cc901566d-workload-socket\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl\" (UID: \"4c7f07f0-82ce-4e5b-97c5-946cc901566d\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl"
Nov 01 05:30:52.171187 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:52.171082    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c7f07f0-82ce-4e5b-97c5-946cc901566d-istiod-ca-cert\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl\" (UID: \"4c7f07f0-82ce-4e5b-97c5-946cc901566d\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl"
Nov 01 05:30:52.171187 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:52.171141    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c7f07f0-82ce-4e5b-97c5-946cc901566d-workload-certs\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl\" (UID: \"4c7f07f0-82ce-4e5b-97c5-946cc901566d\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl"
Nov 01 05:30:52.171369 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:52.171187    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c7f07f0-82ce-4e5b-97c5-946cc901566d-istio-token\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl\" (UID: \"4c7f07f0-82ce-4e5b-97c5-946cc901566d\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl"
Nov 01 05:30:52.171369 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:52.171265    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c7f07f0-82ce-4e5b-97c5-946cc901566d-istio-envoy\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl\" (UID: \"4c7f07f0-82ce-4e5b-97c5-946cc901566d\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl"
Nov 01 05:30:52.171369 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:52.171336    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c7f07f0-82ce-4e5b-97c5-946cc901566d-credential-socket\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl\" (UID: \"4c7f07f0-82ce-4e5b-97c5-946cc901566d\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl"
Nov 01 05:30:52.171553 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:52.171393    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c7f07f0-82ce-4e5b-97c5-946cc901566d-istio-podinfo\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl\" (UID: \"4c7f07f0-82ce-4e5b-97c5-946cc901566d\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl"
Nov 01 05:30:52.171553 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:52.171436    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c7f07f0-82ce-4e5b-97c5-946cc901566d-istio-data\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl\" (UID: \"4c7f07f0-82ce-4e5b-97c5-946cc901566d\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl"
Nov 01 05:30:52.171698 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:52.171548    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c7f07f0-82ce-4e5b-97c5-946cc901566d-workload-socket\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl\" (UID: \"4c7f07f0-82ce-4e5b-97c5-946cc901566d\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl"
Nov 01 05:30:52.171698 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:52.171626    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c7f07f0-82ce-4e5b-97c5-946cc901566d-workload-certs\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl\" (UID: \"4c7f07f0-82ce-4e5b-97c5-946cc901566d\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl"
Nov 01 05:30:52.171860 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:52.171766    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c7f07f0-82ce-4e5b-97c5-946cc901566d-istio-data\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl\" (UID: \"4c7f07f0-82ce-4e5b-97c5-946cc901566d\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl"
Nov 01 05:30:52.171937 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:52.171914    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c7f07f0-82ce-4e5b-97c5-946cc901566d-istiod-ca-cert\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl\" (UID: \"4c7f07f0-82ce-4e5b-97c5-946cc901566d\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl"
Nov 01 05:30:52.171990 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:52.171913    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c7f07f0-82ce-4e5b-97c5-946cc901566d-credential-socket\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl\" (UID: \"4c7f07f0-82ce-4e5b-97c5-946cc901566d\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl"
Nov 01 05:30:52.174112 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:52.174085    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c7f07f0-82ce-4e5b-97c5-946cc901566d-istio-envoy\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl\" (UID: \"4c7f07f0-82ce-4e5b-97c5-946cc901566d\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl"
Nov 01 05:30:52.174291 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:52.174272    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c7f07f0-82ce-4e5b-97c5-946cc901566d-istio-podinfo\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl\" (UID: \"4c7f07f0-82ce-4e5b-97c5-946cc901566d\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl"
Nov 01 05:30:52.232800 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:52.232774    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c7f07f0-82ce-4e5b-97c5-946cc901566d-istio-token\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl\" (UID: \"4c7f07f0-82ce-4e5b-97c5-946cc901566d\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl"
Nov 01 05:30:52.232929 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:52.232832    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wpdk\" (UniqueName: \"kubernetes.io/projected/4c7f07f0-82ce-4e5b-97c5-946cc901566d-kube-api-access-8wpdk\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl\" (UID: \"4c7f07f0-82ce-4e5b-97c5-946cc901566d\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl"
Nov 01 05:30:52.240702 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:52.240687    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl"
Nov 01 05:30:52.671404 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:52.671379    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl"]
Nov 01 05:30:52.673566 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:30:52.673537    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c7f07f0_82ce_4e5b_97c5_946cc901566d.slice/crio-fd39bb03e5df69ec8c727258ca5b8e1ac560f546081a2cb2b483c66ede5b30c9 WatchSource:0}: Error finding container fd39bb03e5df69ec8c727258ca5b8e1ac560f546081a2cb2b483c66ede5b30c9: Status 404 returned error can't find the container with id fd39bb03e5df69ec8c727258ca5b8e1ac560f546081a2cb2b483c66ede5b30c9
Nov 01 05:30:52.675995 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:52.675962    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:30:52.676078 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:52.676028    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:30:52.676078 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:52.676056    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:30:52.930257 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:52.930170    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl" event={"ID":"4c7f07f0-82ce-4e5b-97c5-946cc901566d","Type":"ContainerStarted","Data":"a6e7821944ee9614d85fc9b232cf0f63caa1e755014ab9f1eb2029146e0d9415"}
Nov 01 05:30:52.930257 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:52.930236    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl" event={"ID":"4c7f07f0-82ce-4e5b-97c5-946cc901566d","Type":"ContainerStarted","Data":"fd39bb03e5df69ec8c727258ca5b8e1ac560f546081a2cb2b483c66ede5b30c9"}
Nov 01 05:30:53.051823 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:53.051763    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl" podStartSLOduration=2.051744298 podStartE2EDuration="2.051744298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:30:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:30:53.049128265 +0000 UTC m=+1631.119925018" watchObservedRunningTime="2026-11-01 05:30:53.051744298 +0000 UTC m=+1631.122541019"
Nov 01 05:30:53.241794 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:53.241713    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl"
Nov 01 05:30:53.243294 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:53.243269    5689 patch_prober.go:29] interesting pod/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Nov 01 05:30:53.243411 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:53.243317    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl" podUID="4c7f07f0-82ce-4e5b-97c5-946cc901566d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Nov 01 05:30:54.241137 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:54.241098    5689 patch_prober.go:29] interesting pod/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Nov 01 05:30:54.241559 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:54.241168    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl" podUID="4c7f07f0-82ce-4e5b-97c5-946cc901566d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Nov 01 05:30:54.433996 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:54.433964    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:30:54.707975 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:54.707935    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5lvz97" podUID="6775cee0-3876-46a8-b8f8-53b420c8277e" containerName="payload-processing" containerID="cri-o://47c1de4d5ca2711b84e2f129a93f2a2a66fc0a64d6e21e260878172d921d3739" gracePeriod=25
Nov 01 05:30:54.947106 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:54.947071    5689 generic.go:356] "Generic (PLEG): container finished" podID="6775cee0-3876-46a8-b8f8-53b420c8277e" containerID="47c1de4d5ca2711b84e2f129a93f2a2a66fc0a64d6e21e260878172d921d3739" exitCode=0
Nov 01 05:30:54.947802 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:54.947779    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5lvz97" event={"ID":"6775cee0-3876-46a8-b8f8-53b420c8277e","Type":"ContainerDied","Data":"47c1de4d5ca2711b84e2f129a93f2a2a66fc0a64d6e21e260878172d921d3739"}
Nov 01 05:30:55.029576 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:55.029552    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5lvz97"
Nov 01 05:30:55.127935 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:55.127901    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6775cee0-3876-46a8-b8f8-53b420c8277e-kube-api-access-lrhtj\" (UniqueName: \"kubernetes.io/projected/6775cee0-3876-46a8-b8f8-53b420c8277e-kube-api-access-lrhtj\") pod \"6775cee0-3876-46a8-b8f8-53b420c8277e\" (UID: \"6775cee0-3876-46a8-b8f8-53b420c8277e\") "
Nov 01 05:30:55.128069 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:55.128045    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6775cee0-3876-46a8-b8f8-53b420c8277e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6775cee0-3876-46a8-b8f8-53b420c8277e-routing-overlay-volume\") pod \"6775cee0-3876-46a8-b8f8-53b420c8277e\" (UID: \"6775cee0-3876-46a8-b8f8-53b420c8277e\") "
Nov 01 05:30:55.128178 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:55.128104    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6775cee0-3876-46a8-b8f8-53b420c8277e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6775cee0-3876-46a8-b8f8-53b420c8277e-provider-credentials\") pod \"6775cee0-3876-46a8-b8f8-53b420c8277e\" (UID: \"6775cee0-3876-46a8-b8f8-53b420c8277e\") "
Nov 01 05:30:55.128239 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:55.128196    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6775cee0-3876-46a8-b8f8-53b420c8277e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6775cee0-3876-46a8-b8f8-53b420c8277e-plugins-config-volume\") pod \"6775cee0-3876-46a8-b8f8-53b420c8277e\" (UID: \"6775cee0-3876-46a8-b8f8-53b420c8277e\") "
Nov 01 05:30:55.128419 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:55.128389    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6775cee0-3876-46a8-b8f8-53b420c8277e-routing-overlay-volume" pod "6775cee0-3876-46a8-b8f8-53b420c8277e" (UID: "6775cee0-3876-46a8-b8f8-53b420c8277e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:30:55.128599 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:55.128577    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6775cee0-3876-46a8-b8f8-53b420c8277e-plugins-config-volume" pod "6775cee0-3876-46a8-b8f8-53b420c8277e" (UID: "6775cee0-3876-46a8-b8f8-53b420c8277e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:30:55.128776 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:55.128755    5689 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6775cee0-3876-46a8-b8f8-53b420c8277e-plugins-config-volume\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:30:55.128843 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:55.128786    5689 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6775cee0-3876-46a8-b8f8-53b420c8277e-routing-overlay-volume\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:30:55.130796 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:55.130765    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6775cee0-3876-46a8-b8f8-53b420c8277e-kube-api-access-lrhtj" pod "6775cee0-3876-46a8-b8f8-53b420c8277e" (UID: "6775cee0-3876-46a8-b8f8-53b420c8277e"). InnerVolumeSpecName "kube-api-access-lrhtj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:30:55.130887 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:55.130791    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6775cee0-3876-46a8-b8f8-53b420c8277e-provider-credentials" pod "6775cee0-3876-46a8-b8f8-53b420c8277e" (UID: "6775cee0-3876-46a8-b8f8-53b420c8277e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:30:55.229726 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:55.229693    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrhtj\" (UniqueName: \"kubernetes.io/projected/6775cee0-3876-46a8-b8f8-53b420c8277e-kube-api-access-lrhtj\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:30:55.229726 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:55.229720    5689 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6775cee0-3876-46a8-b8f8-53b420c8277e-provider-credentials\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:30:55.241212 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:55.241185    5689 patch_prober.go:29] interesting pod/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Nov 01 05:30:55.241579 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:55.241232    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl" podUID="4c7f07f0-82ce-4e5b-97c5-946cc901566d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Nov 01 05:30:55.955224 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:55.955190    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5lvz97"
Nov 01 05:30:55.955429 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:55.955229    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5lvz97" event={"ID":"6775cee0-3876-46a8-b8f8-53b420c8277e","Type":"ContainerDied","Data":"c5f24bfb5da1707d150249fc6b9ca4f4478ad614cb038b2fe7b10e60612c2ec6"}
Nov 01 05:30:55.955429 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:55.955303    5689 scope.go:122] "RemoveContainer" containerID="47c1de4d5ca2711b84e2f129a93f2a2a66fc0a64d6e21e260878172d921d3739"
Nov 01 05:30:56.119479 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:56.119437    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5lvz97"]
Nov 01 05:30:56.169041 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:56.169011    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-6bf5lvz97"]
Nov 01 05:30:56.241426 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:56.241341    5689 patch_prober.go:29] interesting pod/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Nov 01 05:30:56.241426 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:56.241417    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl" podUID="4c7f07f0-82ce-4e5b-97c5-946cc901566d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Nov 01 05:30:57.241769 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:57.241731    5689 patch_prober.go:29] interesting pod/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Nov 01 05:30:57.242183 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:57.241785    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl" podUID="4c7f07f0-82ce-4e5b-97c5-946cc901566d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Nov 01 05:30:58.067645 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:58.067615    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6775cee0-3876-46a8-b8f8-53b420c8277e" path="/var/lib/kubelet/pods/6775cee0-3876-46a8-b8f8-53b420c8277e/volumes"
Nov 01 05:30:58.241232 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:58.241196    5689 patch_prober.go:29] interesting pod/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Nov 01 05:30:58.241400 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:58.241264    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl" podUID="4c7f07f0-82ce-4e5b-97c5-946cc901566d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Nov 01 05:30:59.241367 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:59.241336    5689 patch_prober.go:29] interesting pod/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Nov 01 05:30:59.241781 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:59.241387    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl" podUID="4c7f07f0-82ce-4e5b-97c5-946cc901566d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Nov 01 05:30:59.338522 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:30:59.333922    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:31:00.069352 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:31:00.069321    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:31:00.246326 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:00.246297    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl"
Nov 01 05:31:00.246771 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:00.246565    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl"
Nov 01 05:31:00.247293 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:00.247274    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl"
Nov 01 05:31:02.162103 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:02.162072    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:31:02.905061 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:02.905017    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:31:04.183474 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:04.183095    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:31:09.022529 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:09.022502    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:31:13.817357 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:13.817321    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:31:15.303189 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:15.303158    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:31:15.502479 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:15.499346    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:31:16.426706 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:16.426677    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:31:17.762797 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:17.762769    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:31:18.897228 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:18.897199    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:31:19.085497 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:19.085433    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:31:23.915765 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:23.915735    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:31:24.096359 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:24.096328    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:31:28.342877 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:28.342844    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:31:29.641227 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:29.641200    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr"]
Nov 01 05:31:29.641701 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:29.641402    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr" podUID="54adae87-71b3-4dfa-8eb0-3ad86c47e5a8" containerName="istio-proxy" containerID="cri-o://da06ff24d08604bb2d917b84c2b344bab48dc489f548c0386cf59c466f69b16e" gracePeriod=30
Nov 01 05:31:30.068525 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:31:30.068477    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:31:30.220397 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:30.220338    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:31:30.398435 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:30.398410    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:31:33.906545 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:33.906504    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-z24p2"]
Nov 01 05:31:33.907036 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:33.906747    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-z24p2" podUID="ad96da25-2a2b-471c-9a7c-2c3f166553de" containerName="istio-proxy" containerID="cri-o://3ffc60f3476870f8de51f34f26fc3c7a366ba32d34c26d5823d3be67307b7b19" gracePeriod=30
Nov 01 05:31:33.955689 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:33.955658    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4s7z6"]
Nov 01 05:31:33.955914 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:33.955876    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4s7z6" podUID="b68feecd-350f-4582-8cca-c966f065e929" containerName="istio-proxy" containerID="cri-o://2c6c8a5d3577de09ff6c5bffc2e59004fd58614bdd4422ee1314d4a62fc5c716" gracePeriod=30
Nov 01 05:31:33.980031 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:33.980002    5689 patch_prober.go:29] interesting pod/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 05:31:33.980178 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:33.980070    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr" podUID="54adae87-71b3-4dfa-8eb0-3ad86c47e5a8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 05:31:34.138310 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:34.138281    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-d989mcgpt"]
Nov 01 05:31:34.902867 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:34.902842    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr"
Nov 01 05:31:34.939043 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:34.938966    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54adae87-71b3-4dfa-8eb0-3ad86c47e5a8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54adae87-71b3-4dfa-8eb0-3ad86c47e5a8-workload-certs\") pod \"54adae87-71b3-4dfa-8eb0-3ad86c47e5a8\" (UID: \"54adae87-71b3-4dfa-8eb0-3ad86c47e5a8\") "
Nov 01 05:31:34.939043 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:34.939027    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54adae87-71b3-4dfa-8eb0-3ad86c47e5a8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54adae87-71b3-4dfa-8eb0-3ad86c47e5a8-istio-data\") pod \"54adae87-71b3-4dfa-8eb0-3ad86c47e5a8\" (UID: \"54adae87-71b3-4dfa-8eb0-3ad86c47e5a8\") "
Nov 01 05:31:34.939588 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:34.939066    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54adae87-71b3-4dfa-8eb0-3ad86c47e5a8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54adae87-71b3-4dfa-8eb0-3ad86c47e5a8-istio-envoy\") pod \"54adae87-71b3-4dfa-8eb0-3ad86c47e5a8\" (UID: \"54adae87-71b3-4dfa-8eb0-3ad86c47e5a8\") "
Nov 01 05:31:34.939588 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:34.939099    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54adae87-71b3-4dfa-8eb0-3ad86c47e5a8-istio-token\" (UniqueName: \"kubernetes.io/projected/54adae87-71b3-4dfa-8eb0-3ad86c47e5a8-istio-token\") pod \"54adae87-71b3-4dfa-8eb0-3ad86c47e5a8\" (UID: \"54adae87-71b3-4dfa-8eb0-3ad86c47e5a8\") "
Nov 01 05:31:34.939588 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:34.939149    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/54adae87-71b3-4dfa-8eb0-3ad86c47e5a8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54adae87-71b3-4dfa-8eb0-3ad86c47e5a8-istio-podinfo\") pod \"54adae87-71b3-4dfa-8eb0-3ad86c47e5a8\" (UID: \"54adae87-71b3-4dfa-8eb0-3ad86c47e5a8\") "
Nov 01 05:31:34.939588 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:34.939206    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54adae87-71b3-4dfa-8eb0-3ad86c47e5a8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54adae87-71b3-4dfa-8eb0-3ad86c47e5a8-workload-socket\") pod \"54adae87-71b3-4dfa-8eb0-3ad86c47e5a8\" (UID: \"54adae87-71b3-4dfa-8eb0-3ad86c47e5a8\") "
Nov 01 05:31:34.939588 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:34.939279    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54adae87-71b3-4dfa-8eb0-3ad86c47e5a8-workload-certs" pod "54adae87-71b3-4dfa-8eb0-3ad86c47e5a8" (UID: "54adae87-71b3-4dfa-8eb0-3ad86c47e5a8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:31:34.939588 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:34.939326    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54adae87-71b3-4dfa-8eb0-3ad86c47e5a8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54adae87-71b3-4dfa-8eb0-3ad86c47e5a8-credential-socket\") pod \"54adae87-71b3-4dfa-8eb0-3ad86c47e5a8\" (UID: \"54adae87-71b3-4dfa-8eb0-3ad86c47e5a8\") "
Nov 01 05:31:34.939588 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:34.939390    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54adae87-71b3-4dfa-8eb0-3ad86c47e5a8-kube-api-access-mzsvx\" (UniqueName: \"kubernetes.io/projected/54adae87-71b3-4dfa-8eb0-3ad86c47e5a8-kube-api-access-mzsvx\") pod \"54adae87-71b3-4dfa-8eb0-3ad86c47e5a8\" (UID: \"54adae87-71b3-4dfa-8eb0-3ad86c47e5a8\") "
Nov 01 05:31:34.939588 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:34.939401    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54adae87-71b3-4dfa-8eb0-3ad86c47e5a8-istio-data" pod "54adae87-71b3-4dfa-8eb0-3ad86c47e5a8" (UID: "54adae87-71b3-4dfa-8eb0-3ad86c47e5a8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:31:34.939588 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:34.939495    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54adae87-71b3-4dfa-8eb0-3ad86c47e5a8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54adae87-71b3-4dfa-8eb0-3ad86c47e5a8-istiod-ca-cert\") pod \"54adae87-71b3-4dfa-8eb0-3ad86c47e5a8\" (UID: \"54adae87-71b3-4dfa-8eb0-3ad86c47e5a8\") "
Nov 01 05:31:34.939588 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:34.939517    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54adae87-71b3-4dfa-8eb0-3ad86c47e5a8-workload-socket" pod "54adae87-71b3-4dfa-8eb0-3ad86c47e5a8" (UID: "54adae87-71b3-4dfa-8eb0-3ad86c47e5a8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:31:34.940274 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:34.939594    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54adae87-71b3-4dfa-8eb0-3ad86c47e5a8-credential-socket" pod "54adae87-71b3-4dfa-8eb0-3ad86c47e5a8" (UID: "54adae87-71b3-4dfa-8eb0-3ad86c47e5a8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:31:34.940274 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:34.940000    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54adae87-71b3-4dfa-8eb0-3ad86c47e5a8-istiod-ca-cert" pod "54adae87-71b3-4dfa-8eb0-3ad86c47e5a8" (UID: "54adae87-71b3-4dfa-8eb0-3ad86c47e5a8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:31:34.940274 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:34.940187    5689 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54adae87-71b3-4dfa-8eb0-3ad86c47e5a8-workload-certs\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:31:34.940274 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:34.940240    5689 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54adae87-71b3-4dfa-8eb0-3ad86c47e5a8-istio-data\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:31:34.940274 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:34.940255    5689 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54adae87-71b3-4dfa-8eb0-3ad86c47e5a8-workload-socket\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:31:34.940274 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:34.940271    5689 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54adae87-71b3-4dfa-8eb0-3ad86c47e5a8-credential-socket\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:31:34.941632 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:34.941608    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54adae87-71b3-4dfa-8eb0-3ad86c47e5a8-istio-token" pod "54adae87-71b3-4dfa-8eb0-3ad86c47e5a8" (UID: "54adae87-71b3-4dfa-8eb0-3ad86c47e5a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:31:34.941750 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:34.941631    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/54adae87-71b3-4dfa-8eb0-3ad86c47e5a8-istio-podinfo" pod "54adae87-71b3-4dfa-8eb0-3ad86c47e5a8" (UID: "54adae87-71b3-4dfa-8eb0-3ad86c47e5a8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 05:31:34.941750 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:34.941722    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54adae87-71b3-4dfa-8eb0-3ad86c47e5a8-kube-api-access-mzsvx" pod "54adae87-71b3-4dfa-8eb0-3ad86c47e5a8" (UID: "54adae87-71b3-4dfa-8eb0-3ad86c47e5a8"). InnerVolumeSpecName "kube-api-access-mzsvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:31:34.941750 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:34.941724    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54adae87-71b3-4dfa-8eb0-3ad86c47e5a8-istio-envoy" pod "54adae87-71b3-4dfa-8eb0-3ad86c47e5a8" (UID: "54adae87-71b3-4dfa-8eb0-3ad86c47e5a8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:31:35.058762 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:35.058619    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:31:35.060805 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:35.060775    5689 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54adae87-71b3-4dfa-8eb0-3ad86c47e5a8-istio-envoy\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:31:35.060805 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:35.060809    5689 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54adae87-71b3-4dfa-8eb0-3ad86c47e5a8-istio-token\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:31:35.061010 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:35.060823    5689 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54adae87-71b3-4dfa-8eb0-3ad86c47e5a8-istio-podinfo\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:31:35.061010 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:35.060836    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzsvx\" (UniqueName: \"kubernetes.io/projected/54adae87-71b3-4dfa-8eb0-3ad86c47e5a8-kube-api-access-mzsvx\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:31:35.061010 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:35.060849    5689 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54adae87-71b3-4dfa-8eb0-3ad86c47e5a8-istiod-ca-cert\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:31:35.323689 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:35.323598    5689 generic.go:356] "Generic (PLEG): container finished" podID="54adae87-71b3-4dfa-8eb0-3ad86c47e5a8" containerID="da06ff24d08604bb2d917b84c2b344bab48dc489f548c0386cf59c466f69b16e" exitCode=0
Nov 01 05:31:35.323689 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:35.323646    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr"
Nov 01 05:31:35.323689 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:35.323679    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr" event={"ID":"54adae87-71b3-4dfa-8eb0-3ad86c47e5a8","Type":"ContainerDied","Data":"da06ff24d08604bb2d917b84c2b344bab48dc489f548c0386cf59c466f69b16e"}
Nov 01 05:31:35.323929 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:35.323717    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr" event={"ID":"54adae87-71b3-4dfa-8eb0-3ad86c47e5a8","Type":"ContainerDied","Data":"4b1c0fb2639165866065c303ba3166e59f1452fec7ed5dac66d3b4af9464a968"}
Nov 01 05:31:35.323929 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:35.323738    5689 scope.go:122] "RemoveContainer" containerID="da06ff24d08604bb2d917b84c2b344bab48dc489f548c0386cf59c466f69b16e"
Nov 01 05:31:35.336134 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:35.336116    5689 scope.go:122] "RemoveContainer" containerID="da06ff24d08604bb2d917b84c2b344bab48dc489f548c0386cf59c466f69b16e"
Nov 01 05:31:35.336575 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:31:35.336538    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da06ff24d08604bb2d917b84c2b344bab48dc489f548c0386cf59c466f69b16e\": container with ID starting with da06ff24d08604bb2d917b84c2b344bab48dc489f548c0386cf59c466f69b16e not found: ID does not exist" containerID="da06ff24d08604bb2d917b84c2b344bab48dc489f548c0386cf59c466f69b16e"
Nov 01 05:31:35.336720 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:35.336586    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da06ff24d08604bb2d917b84c2b344bab48dc489f548c0386cf59c466f69b16e"} err="failed to get container status \"da06ff24d08604bb2d917b84c2b344bab48dc489f548c0386cf59c466f69b16e\": rpc error: code = NotFound desc = could not find container \"da06ff24d08604bb2d917b84c2b344bab48dc489f548c0386cf59c466f69b16e\": container with ID starting with da06ff24d08604bb2d917b84c2b344bab48dc489f548c0386cf59c466f69b16e not found: ID does not exist"
Nov 01 05:31:35.489796 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:35.489760    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr"]
Nov 01 05:31:35.539182 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:35.539151    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-pgpkr"]
Nov 01 05:31:36.067816 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:36.067787    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54adae87-71b3-4dfa-8eb0-3ad86c47e5a8" path="/var/lib/kubelet/pods/54adae87-71b3-4dfa-8eb0-3ad86c47e5a8/volumes"
Nov 01 05:31:36.703416 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:36.703383    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:31:36.887777 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:36.887740    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:31:37.229716 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:37.229682    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-9db8tm77q"]
Nov 01 05:31:37.652651 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:37.652620    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:31:38.128346 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:38.128271    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:31:38.340107 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:38.340008    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:31:38.679531 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:38.679498    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:31:38.864872 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:38.864840    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:31:39.138774 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.138729    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-d989mcgpt" podUID="494e7554-111b-4033-8a49-68ae0ad86c0b" containerName="payload-processing" containerID="cri-o://22fae2f9c602d3da8b94d4c8ee15075b806608777b8079a03711867bd5ceaf67" gracePeriod=25
Nov 01 05:31:39.251422 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.251372    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-z24p2"
Nov 01 05:31:39.254506 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.254487    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4s7z6"
Nov 01 05:31:39.293603 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.293574    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ad96da25-2a2b-471c-9a7c-2c3f166553de-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad96da25-2a2b-471c-9a7c-2c3f166553de-istio-podinfo\") pod \"ad96da25-2a2b-471c-9a7c-2c3f166553de\" (UID: \"ad96da25-2a2b-471c-9a7c-2c3f166553de\") "
Nov 01 05:31:39.293769 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.293624    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad96da25-2a2b-471c-9a7c-2c3f166553de-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad96da25-2a2b-471c-9a7c-2c3f166553de-istiod-ca-cert\") pod \"ad96da25-2a2b-471c-9a7c-2c3f166553de\" (UID: \"ad96da25-2a2b-471c-9a7c-2c3f166553de\") "
Nov 01 05:31:39.293769 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.293656    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b68feecd-350f-4582-8cca-c966f065e929-istio-token\" (UniqueName: \"kubernetes.io/projected/b68feecd-350f-4582-8cca-c966f065e929-istio-token\") pod \"b68feecd-350f-4582-8cca-c966f065e929\" (UID: \"b68feecd-350f-4582-8cca-c966f065e929\") "
Nov 01 05:31:39.293769 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.293679    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad96da25-2a2b-471c-9a7c-2c3f166553de-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad96da25-2a2b-471c-9a7c-2c3f166553de-istio-data\") pod \"ad96da25-2a2b-471c-9a7c-2c3f166553de\" (UID: \"ad96da25-2a2b-471c-9a7c-2c3f166553de\") "
Nov 01 05:31:39.293769 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.293708    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad96da25-2a2b-471c-9a7c-2c3f166553de-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad96da25-2a2b-471c-9a7c-2c3f166553de-credential-socket\") pod \"ad96da25-2a2b-471c-9a7c-2c3f166553de\" (UID: \"ad96da25-2a2b-471c-9a7c-2c3f166553de\") "
Nov 01 05:31:39.293769 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.293768    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad96da25-2a2b-471c-9a7c-2c3f166553de-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad96da25-2a2b-471c-9a7c-2c3f166553de-workload-socket\") pod \"ad96da25-2a2b-471c-9a7c-2c3f166553de\" (UID: \"ad96da25-2a2b-471c-9a7c-2c3f166553de\") "
Nov 01 05:31:39.294032 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.293826    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b68feecd-350f-4582-8cca-c966f065e929-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b68feecd-350f-4582-8cca-c966f065e929-istio-data\") pod \"b68feecd-350f-4582-8cca-c966f065e929\" (UID: \"b68feecd-350f-4582-8cca-c966f065e929\") "
Nov 01 05:31:39.294032 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.293889    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b68feecd-350f-4582-8cca-c966f065e929-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b68feecd-350f-4582-8cca-c966f065e929-workload-socket\") pod \"b68feecd-350f-4582-8cca-c966f065e929\" (UID: \"b68feecd-350f-4582-8cca-c966f065e929\") "
Nov 01 05:31:39.294032 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.293934    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b68feecd-350f-4582-8cca-c966f065e929-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b68feecd-350f-4582-8cca-c966f065e929-workload-certs\") pod \"b68feecd-350f-4582-8cca-c966f065e929\" (UID: \"b68feecd-350f-4582-8cca-c966f065e929\") "
Nov 01 05:31:39.294032 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.293967    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b68feecd-350f-4582-8cca-c966f065e929-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b68feecd-350f-4582-8cca-c966f065e929-istiod-ca-cert\") pod \"b68feecd-350f-4582-8cca-c966f065e929\" (UID: \"b68feecd-350f-4582-8cca-c966f065e929\") "
Nov 01 05:31:39.294032 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.293996    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b68feecd-350f-4582-8cca-c966f065e929-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b68feecd-350f-4582-8cca-c966f065e929-credential-socket\") pod \"b68feecd-350f-4582-8cca-c966f065e929\" (UID: \"b68feecd-350f-4582-8cca-c966f065e929\") "
Nov 01 05:31:39.294283 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.294042    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b68feecd-350f-4582-8cca-c966f065e929-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b68feecd-350f-4582-8cca-c966f065e929-istio-podinfo\") pod \"b68feecd-350f-4582-8cca-c966f065e929\" (UID: \"b68feecd-350f-4582-8cca-c966f065e929\") "
Nov 01 05:31:39.294283 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.294148    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad96da25-2a2b-471c-9a7c-2c3f166553de-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad96da25-2a2b-471c-9a7c-2c3f166553de-workload-certs\") pod \"ad96da25-2a2b-471c-9a7c-2c3f166553de\" (UID: \"ad96da25-2a2b-471c-9a7c-2c3f166553de\") "
Nov 01 05:31:39.294283 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.294192    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad96da25-2a2b-471c-9a7c-2c3f166553de-kube-api-access-gf4wt\" (UniqueName: \"kubernetes.io/projected/ad96da25-2a2b-471c-9a7c-2c3f166553de-kube-api-access-gf4wt\") pod \"ad96da25-2a2b-471c-9a7c-2c3f166553de\" (UID: \"ad96da25-2a2b-471c-9a7c-2c3f166553de\") "
Nov 01 05:31:39.294283 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.294252    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad96da25-2a2b-471c-9a7c-2c3f166553de-istio-token\" (UniqueName: \"kubernetes.io/projected/ad96da25-2a2b-471c-9a7c-2c3f166553de-istio-token\") pod \"ad96da25-2a2b-471c-9a7c-2c3f166553de\" (UID: \"ad96da25-2a2b-471c-9a7c-2c3f166553de\") "
Nov 01 05:31:39.294818 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.294293    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b68feecd-350f-4582-8cca-c966f065e929-kube-api-access-hw55p\" (UniqueName: \"kubernetes.io/projected/b68feecd-350f-4582-8cca-c966f065e929-kube-api-access-hw55p\") pod \"b68feecd-350f-4582-8cca-c966f065e929\" (UID: \"b68feecd-350f-4582-8cca-c966f065e929\") "
Nov 01 05:31:39.294818 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.294328    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b68feecd-350f-4582-8cca-c966f065e929-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b68feecd-350f-4582-8cca-c966f065e929-istio-envoy\") pod \"b68feecd-350f-4582-8cca-c966f065e929\" (UID: \"b68feecd-350f-4582-8cca-c966f065e929\") "
Nov 01 05:31:39.294818 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.294370    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad96da25-2a2b-471c-9a7c-2c3f166553de-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad96da25-2a2b-471c-9a7c-2c3f166553de-istio-envoy\") pod \"ad96da25-2a2b-471c-9a7c-2c3f166553de\" (UID: \"ad96da25-2a2b-471c-9a7c-2c3f166553de\") "
Nov 01 05:31:39.294818 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.293992    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad96da25-2a2b-471c-9a7c-2c3f166553de-istio-data" pod "ad96da25-2a2b-471c-9a7c-2c3f166553de" (UID: "ad96da25-2a2b-471c-9a7c-2c3f166553de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:31:39.294818 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.294102    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad96da25-2a2b-471c-9a7c-2c3f166553de-istiod-ca-cert" pod "ad96da25-2a2b-471c-9a7c-2c3f166553de" (UID: "ad96da25-2a2b-471c-9a7c-2c3f166553de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:31:39.294818 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.294185    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b68feecd-350f-4582-8cca-c966f065e929-istio-data" pod "b68feecd-350f-4582-8cca-c966f065e929" (UID: "b68feecd-350f-4582-8cca-c966f065e929"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:31:39.294818 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.294196    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b68feecd-350f-4582-8cca-c966f065e929-workload-certs" pod "b68feecd-350f-4582-8cca-c966f065e929" (UID: "b68feecd-350f-4582-8cca-c966f065e929"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:31:39.294818 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.294373    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b68feecd-350f-4582-8cca-c966f065e929-credential-socket" pod "b68feecd-350f-4582-8cca-c966f065e929" (UID: "b68feecd-350f-4582-8cca-c966f065e929"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:31:39.294818 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.294386    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad96da25-2a2b-471c-9a7c-2c3f166553de-credential-socket" pod "ad96da25-2a2b-471c-9a7c-2c3f166553de" (UID: "ad96da25-2a2b-471c-9a7c-2c3f166553de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:31:39.294818 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.294392    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b68feecd-350f-4582-8cca-c966f065e929-workload-socket" pod "b68feecd-350f-4582-8cca-c966f065e929" (UID: "b68feecd-350f-4582-8cca-c966f065e929"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:31:39.294818 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.294601    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad96da25-2a2b-471c-9a7c-2c3f166553de-workload-socket" pod "ad96da25-2a2b-471c-9a7c-2c3f166553de" (UID: "ad96da25-2a2b-471c-9a7c-2c3f166553de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:31:39.294818 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.294683    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b68feecd-350f-4582-8cca-c966f065e929-istiod-ca-cert" pod "b68feecd-350f-4582-8cca-c966f065e929" (UID: "b68feecd-350f-4582-8cca-c966f065e929"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:31:39.295709 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.294994    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad96da25-2a2b-471c-9a7c-2c3f166553de-workload-certs" pod "ad96da25-2a2b-471c-9a7c-2c3f166553de" (UID: "ad96da25-2a2b-471c-9a7c-2c3f166553de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:31:39.298357 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.297784    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad96da25-2a2b-471c-9a7c-2c3f166553de-istio-token" pod "ad96da25-2a2b-471c-9a7c-2c3f166553de" (UID: "ad96da25-2a2b-471c-9a7c-2c3f166553de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:31:39.298357 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.298129    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b68feecd-350f-4582-8cca-c966f065e929-istio-envoy" pod "b68feecd-350f-4582-8cca-c966f065e929" (UID: "b68feecd-350f-4582-8cca-c966f065e929"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:31:39.298357 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.298194    5689 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad96da25-2a2b-471c-9a7c-2c3f166553de-istio-token\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:31:39.298357 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.298212    5689 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad96da25-2a2b-471c-9a7c-2c3f166553de-istiod-ca-cert\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:31:39.298357 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.298227    5689 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad96da25-2a2b-471c-9a7c-2c3f166553de-istio-data\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:31:39.298357 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.298241    5689 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad96da25-2a2b-471c-9a7c-2c3f166553de-credential-socket\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:31:39.298357 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.298255    5689 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad96da25-2a2b-471c-9a7c-2c3f166553de-workload-socket\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:31:39.298357 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.298269    5689 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b68feecd-350f-4582-8cca-c966f065e929-istio-data\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:31:39.298357 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.298283    5689 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b68feecd-350f-4582-8cca-c966f065e929-workload-socket\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:31:39.298357 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.298296    5689 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b68feecd-350f-4582-8cca-c966f065e929-workload-certs\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:31:39.298357 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.298308    5689 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b68feecd-350f-4582-8cca-c966f065e929-istiod-ca-cert\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:31:39.298357 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.298322    5689 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b68feecd-350f-4582-8cca-c966f065e929-credential-socket\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:31:39.298357 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.298334    5689 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad96da25-2a2b-471c-9a7c-2c3f166553de-workload-certs\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:31:39.299636 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.299582    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad96da25-2a2b-471c-9a7c-2c3f166553de-istio-envoy" pod "ad96da25-2a2b-471c-9a7c-2c3f166553de" (UID: "ad96da25-2a2b-471c-9a7c-2c3f166553de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:31:39.300249 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.300227    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b68feecd-350f-4582-8cca-c966f065e929-istio-token" pod "b68feecd-350f-4582-8cca-c966f065e929" (UID: "b68feecd-350f-4582-8cca-c966f065e929"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:31:39.302721 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.302701    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b68feecd-350f-4582-8cca-c966f065e929-kube-api-access-hw55p" pod "b68feecd-350f-4582-8cca-c966f065e929" (UID: "b68feecd-350f-4582-8cca-c966f065e929"). InnerVolumeSpecName "kube-api-access-hw55p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:31:39.302803 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.302792    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b68feecd-350f-4582-8cca-c966f065e929-istio-podinfo" pod "b68feecd-350f-4582-8cca-c966f065e929" (UID: "b68feecd-350f-4582-8cca-c966f065e929"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 05:31:39.302864 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.302852    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad96da25-2a2b-471c-9a7c-2c3f166553de-kube-api-access-gf4wt" pod "ad96da25-2a2b-471c-9a7c-2c3f166553de" (UID: "ad96da25-2a2b-471c-9a7c-2c3f166553de"). InnerVolumeSpecName "kube-api-access-gf4wt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:31:39.304420 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.304401    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ad96da25-2a2b-471c-9a7c-2c3f166553de-istio-podinfo" pod "ad96da25-2a2b-471c-9a7c-2c3f166553de" (UID: "ad96da25-2a2b-471c-9a7c-2c3f166553de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 05:31:39.383677 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.383645    5689 generic.go:356] "Generic (PLEG): container finished" podID="b68feecd-350f-4582-8cca-c966f065e929" containerID="2c6c8a5d3577de09ff6c5bffc2e59004fd58614bdd4422ee1314d4a62fc5c716" exitCode=0
Nov 01 05:31:39.384122 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.383691    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4s7z6"
Nov 01 05:31:39.384122 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.383728    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4s7z6" event={"ID":"b68feecd-350f-4582-8cca-c966f065e929","Type":"ContainerDied","Data":"2c6c8a5d3577de09ff6c5bffc2e59004fd58614bdd4422ee1314d4a62fc5c716"}
Nov 01 05:31:39.384122 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.383800    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4s7z6" event={"ID":"b68feecd-350f-4582-8cca-c966f065e929","Type":"ContainerDied","Data":"bfe6aebabbda4f4c038d5e2492659c04c424a9b3d2eec279106fc13c7ea2ea6b"}
Nov 01 05:31:39.384122 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.383830    5689 scope.go:122] "RemoveContainer" containerID="2c6c8a5d3577de09ff6c5bffc2e59004fd58614bdd4422ee1314d4a62fc5c716"
Nov 01 05:31:39.385686 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.385567    5689 generic.go:356] "Generic (PLEG): container finished" podID="494e7554-111b-4033-8a49-68ae0ad86c0b" containerID="22fae2f9c602d3da8b94d4c8ee15075b806608777b8079a03711867bd5ceaf67" exitCode=0
Nov 01 05:31:39.385686 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.385664    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-d989mcgpt" event={"ID":"494e7554-111b-4033-8a49-68ae0ad86c0b","Type":"ContainerDied","Data":"22fae2f9c602d3da8b94d4c8ee15075b806608777b8079a03711867bd5ceaf67"}
Nov 01 05:31:39.387410 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.387376    5689 generic.go:356] "Generic (PLEG): container finished" podID="ad96da25-2a2b-471c-9a7c-2c3f166553de" containerID="3ffc60f3476870f8de51f34f26fc3c7a366ba32d34c26d5823d3be67307b7b19" exitCode=0
Nov 01 05:31:39.387410 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.387409    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-z24p2" event={"ID":"ad96da25-2a2b-471c-9a7c-2c3f166553de","Type":"ContainerDied","Data":"3ffc60f3476870f8de51f34f26fc3c7a366ba32d34c26d5823d3be67307b7b19"}
Nov 01 05:31:39.387410 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.387417    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-z24p2"
Nov 01 05:31:39.387676 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.387436    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-z24p2" event={"ID":"ad96da25-2a2b-471c-9a7c-2c3f166553de","Type":"ContainerDied","Data":"03aad61553edc1db9440defc857ed4d2b463790abbc0428b2955e36b5ee8db89"}
Nov 01 05:31:39.395768 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.395749    5689 scope.go:122] "RemoveContainer" containerID="2c6c8a5d3577de09ff6c5bffc2e59004fd58614bdd4422ee1314d4a62fc5c716"
Nov 01 05:31:39.396183 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:31:39.396059    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c6c8a5d3577de09ff6c5bffc2e59004fd58614bdd4422ee1314d4a62fc5c716\": container with ID starting with 2c6c8a5d3577de09ff6c5bffc2e59004fd58614bdd4422ee1314d4a62fc5c716 not found: ID does not exist" containerID="2c6c8a5d3577de09ff6c5bffc2e59004fd58614bdd4422ee1314d4a62fc5c716"
Nov 01 05:31:39.396183 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.396098    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c6c8a5d3577de09ff6c5bffc2e59004fd58614bdd4422ee1314d4a62fc5c716"} err="failed to get container status \"2c6c8a5d3577de09ff6c5bffc2e59004fd58614bdd4422ee1314d4a62fc5c716\": rpc error: code = NotFound desc = could not find container \"2c6c8a5d3577de09ff6c5bffc2e59004fd58614bdd4422ee1314d4a62fc5c716\": container with ID starting with 2c6c8a5d3577de09ff6c5bffc2e59004fd58614bdd4422ee1314d4a62fc5c716 not found: ID does not exist"
Nov 01 05:31:39.396183 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.396118    5689 scope.go:122] "RemoveContainer" containerID="3ffc60f3476870f8de51f34f26fc3c7a366ba32d34c26d5823d3be67307b7b19"
Nov 01 05:31:39.399497 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.399476    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gf4wt\" (UniqueName: \"kubernetes.io/projected/ad96da25-2a2b-471c-9a7c-2c3f166553de-kube-api-access-gf4wt\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:31:39.399497 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.399498    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hw55p\" (UniqueName: \"kubernetes.io/projected/b68feecd-350f-4582-8cca-c966f065e929-kube-api-access-hw55p\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:31:39.399649 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.399508    5689 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b68feecd-350f-4582-8cca-c966f065e929-istio-envoy\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:31:39.399649 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.399517    5689 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad96da25-2a2b-471c-9a7c-2c3f166553de-istio-envoy\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:31:39.399649 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.399526    5689 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad96da25-2a2b-471c-9a7c-2c3f166553de-istio-podinfo\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:31:39.399649 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.399535    5689 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b68feecd-350f-4582-8cca-c966f065e929-istio-token\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:31:39.399649 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.399544    5689 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b68feecd-350f-4582-8cca-c966f065e929-istio-podinfo\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:31:39.406358 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.406337    5689 scope.go:122] "RemoveContainer" containerID="3ffc60f3476870f8de51f34f26fc3c7a366ba32d34c26d5823d3be67307b7b19"
Nov 01 05:31:39.406625 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:31:39.406608    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ffc60f3476870f8de51f34f26fc3c7a366ba32d34c26d5823d3be67307b7b19\": container with ID starting with 3ffc60f3476870f8de51f34f26fc3c7a366ba32d34c26d5823d3be67307b7b19 not found: ID does not exist" containerID="3ffc60f3476870f8de51f34f26fc3c7a366ba32d34c26d5823d3be67307b7b19"
Nov 01 05:31:39.406686 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.406631    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ffc60f3476870f8de51f34f26fc3c7a366ba32d34c26d5823d3be67307b7b19"} err="failed to get container status \"3ffc60f3476870f8de51f34f26fc3c7a366ba32d34c26d5823d3be67307b7b19\": rpc error: code = NotFound desc = could not find container \"3ffc60f3476870f8de51f34f26fc3c7a366ba32d34c26d5823d3be67307b7b19\": container with ID starting with 3ffc60f3476870f8de51f34f26fc3c7a366ba32d34c26d5823d3be67307b7b19 not found: ID does not exist"
Nov 01 05:31:39.406984 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.406969    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-d989mcgpt"
Nov 01 05:31:39.500877 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.500797    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/494e7554-111b-4033-8a49-68ae0ad86c0b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/494e7554-111b-4033-8a49-68ae0ad86c0b-routing-overlay-volume\") pod \"494e7554-111b-4033-8a49-68ae0ad86c0b\" (UID: \"494e7554-111b-4033-8a49-68ae0ad86c0b\") "
Nov 01 05:31:39.501024 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.500964    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/494e7554-111b-4033-8a49-68ae0ad86c0b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/494e7554-111b-4033-8a49-68ae0ad86c0b-provider-credentials\") pod \"494e7554-111b-4033-8a49-68ae0ad86c0b\" (UID: \"494e7554-111b-4033-8a49-68ae0ad86c0b\") "
Nov 01 05:31:39.501024 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.500992    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/494e7554-111b-4033-8a49-68ae0ad86c0b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/494e7554-111b-4033-8a49-68ae0ad86c0b-plugins-config-volume\") pod \"494e7554-111b-4033-8a49-68ae0ad86c0b\" (UID: \"494e7554-111b-4033-8a49-68ae0ad86c0b\") "
Nov 01 05:31:39.501152 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.501035    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/494e7554-111b-4033-8a49-68ae0ad86c0b-kube-api-access-7tf5j\" (UniqueName: \"kubernetes.io/projected/494e7554-111b-4033-8a49-68ae0ad86c0b-kube-api-access-7tf5j\") pod \"494e7554-111b-4033-8a49-68ae0ad86c0b\" (UID: \"494e7554-111b-4033-8a49-68ae0ad86c0b\") "
Nov 01 05:31:39.501319 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.501203    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/494e7554-111b-4033-8a49-68ae0ad86c0b-routing-overlay-volume" pod "494e7554-111b-4033-8a49-68ae0ad86c0b" (UID: "494e7554-111b-4033-8a49-68ae0ad86c0b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:31:39.501399 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.501388    5689 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/494e7554-111b-4033-8a49-68ae0ad86c0b-routing-overlay-volume\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:31:39.501437 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.501416    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/494e7554-111b-4033-8a49-68ae0ad86c0b-plugins-config-volume" pod "494e7554-111b-4033-8a49-68ae0ad86c0b" (UID: "494e7554-111b-4033-8a49-68ae0ad86c0b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:31:39.503430 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.503403    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/494e7554-111b-4033-8a49-68ae0ad86c0b-kube-api-access-7tf5j" pod "494e7554-111b-4033-8a49-68ae0ad86c0b" (UID: "494e7554-111b-4033-8a49-68ae0ad86c0b"). InnerVolumeSpecName "kube-api-access-7tf5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:31:39.503430 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.503418    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/494e7554-111b-4033-8a49-68ae0ad86c0b-provider-credentials" pod "494e7554-111b-4033-8a49-68ae0ad86c0b" (UID: "494e7554-111b-4033-8a49-68ae0ad86c0b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:31:39.602504 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.602437    5689 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/494e7554-111b-4033-8a49-68ae0ad86c0b-provider-credentials\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:31:39.602504 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.602498    5689 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/494e7554-111b-4033-8a49-68ae0ad86c0b-plugins-config-volume\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:31:39.602694 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.602516    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tf5j\" (UniqueName: \"kubernetes.io/projected/494e7554-111b-4033-8a49-68ae0ad86c0b-kube-api-access-7tf5j\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:31:39.653245 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.653204    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-z24p2"]
Nov 01 05:31:39.655492 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.655431    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-z24p2"]
Nov 01 05:31:39.832384 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.832314    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4s7z6"]
Nov 01 05:31:39.836553 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:39.836525    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-4s7z6"]
Nov 01 05:31:40.067907 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:40.067867    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad96da25-2a2b-471c-9a7c-2c3f166553de" path="/var/lib/kubelet/pods/ad96da25-2a2b-471c-9a7c-2c3f166553de/volumes"
Nov 01 05:31:40.068385 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:40.068364    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b68feecd-350f-4582-8cca-c966f065e929" path="/var/lib/kubelet/pods/b68feecd-350f-4582-8cca-c966f065e929/volumes"
Nov 01 05:31:40.246214 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:40.246186    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:31:40.396218 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:40.396188    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-d989mcgpt" event={"ID":"494e7554-111b-4033-8a49-68ae0ad86c0b","Type":"ContainerDied","Data":"88323d7c20147730126c3c50ab58f12a85dcebccff10a7139a47044ab62152c9"}
Nov 01 05:31:40.396218 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:40.396212    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-d989mcgpt"
Nov 01 05:31:40.396684 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:40.396240    5689 scope.go:122] "RemoveContainer" containerID="22fae2f9c602d3da8b94d4c8ee15075b806608777b8079a03711867bd5ceaf67"
Nov 01 05:31:40.563463 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:40.563367    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-d989mcgpt"]
Nov 01 05:31:40.567560 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:40.567532    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-d989mcgpt"]
Nov 01 05:31:42.068813 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:42.068782    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="494e7554-111b-4033-8a49-68ae0ad86c0b" path="/var/lib/kubelet/pods/494e7554-111b-4033-8a49-68ae0ad86c0b/volumes"
Nov 01 05:31:42.230903 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:42.230871    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-9db8tm77q" podUID="8c6e7e1d-dfc3-49a1-a6e0-acb6937eecde" containerName="payload-processing" containerID="cri-o://f24976884609d62e9ce0ae6bd4660364b3140fcb062085e4a84e0debb5af47b9" gracePeriod=25
Nov 01 05:31:42.414601 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:42.414566    5689 generic.go:356] "Generic (PLEG): container finished" podID="8c6e7e1d-dfc3-49a1-a6e0-acb6937eecde" containerID="f24976884609d62e9ce0ae6bd4660364b3140fcb062085e4a84e0debb5af47b9" exitCode=0
Nov 01 05:31:42.414765 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:42.414670    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-9db8tm77q" event={"ID":"8c6e7e1d-dfc3-49a1-a6e0-acb6937eecde","Type":"ContainerDied","Data":"f24976884609d62e9ce0ae6bd4660364b3140fcb062085e4a84e0debb5af47b9"}
Nov 01 05:31:42.505407 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:42.505385    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-9db8tm77q"
Nov 01 05:31:42.531887 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:42.531848    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c6e7e1d-dfc3-49a1-a6e0-acb6937eecde-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8c6e7e1d-dfc3-49a1-a6e0-acb6937eecde-routing-overlay-volume\") pod \"8c6e7e1d-dfc3-49a1-a6e0-acb6937eecde\" (UID: \"8c6e7e1d-dfc3-49a1-a6e0-acb6937eecde\") "
Nov 01 05:31:42.532015 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:42.531920    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c6e7e1d-dfc3-49a1-a6e0-acb6937eecde-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8c6e7e1d-dfc3-49a1-a6e0-acb6937eecde-provider-credentials\") pod \"8c6e7e1d-dfc3-49a1-a6e0-acb6937eecde\" (UID: \"8c6e7e1d-dfc3-49a1-a6e0-acb6937eecde\") "
Nov 01 05:31:42.532055 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:42.532010    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c6e7e1d-dfc3-49a1-a6e0-acb6937eecde-kube-api-access-5nz86\" (UniqueName: \"kubernetes.io/projected/8c6e7e1d-dfc3-49a1-a6e0-acb6937eecde-kube-api-access-5nz86\") pod \"8c6e7e1d-dfc3-49a1-a6e0-acb6937eecde\" (UID: \"8c6e7e1d-dfc3-49a1-a6e0-acb6937eecde\") "
Nov 01 05:31:42.532095 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:42.532052    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c6e7e1d-dfc3-49a1-a6e0-acb6937eecde-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c6e7e1d-dfc3-49a1-a6e0-acb6937eecde-plugins-config-volume\") pod \"8c6e7e1d-dfc3-49a1-a6e0-acb6937eecde\" (UID: \"8c6e7e1d-dfc3-49a1-a6e0-acb6937eecde\") "
Nov 01 05:31:42.532176 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:42.532154    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c6e7e1d-dfc3-49a1-a6e0-acb6937eecde-routing-overlay-volume" pod "8c6e7e1d-dfc3-49a1-a6e0-acb6937eecde" (UID: "8c6e7e1d-dfc3-49a1-a6e0-acb6937eecde"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:31:42.532508 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:42.532486    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c6e7e1d-dfc3-49a1-a6e0-acb6937eecde-plugins-config-volume" pod "8c6e7e1d-dfc3-49a1-a6e0-acb6937eecde" (UID: "8c6e7e1d-dfc3-49a1-a6e0-acb6937eecde"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:31:42.532603 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:42.532542    5689 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8c6e7e1d-dfc3-49a1-a6e0-acb6937eecde-routing-overlay-volume\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:31:42.534144 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:42.534119    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c6e7e1d-dfc3-49a1-a6e0-acb6937eecde-provider-credentials" pod "8c6e7e1d-dfc3-49a1-a6e0-acb6937eecde" (UID: "8c6e7e1d-dfc3-49a1-a6e0-acb6937eecde"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:31:42.534235 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:42.534208    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c6e7e1d-dfc3-49a1-a6e0-acb6937eecde-kube-api-access-5nz86" pod "8c6e7e1d-dfc3-49a1-a6e0-acb6937eecde" (UID: "8c6e7e1d-dfc3-49a1-a6e0-acb6937eecde"). InnerVolumeSpecName "kube-api-access-5nz86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:31:42.633787 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:42.633710    5689 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8c6e7e1d-dfc3-49a1-a6e0-acb6937eecde-provider-credentials\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:31:42.633787 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:42.633738    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nz86\" (UniqueName: \"kubernetes.io/projected/8c6e7e1d-dfc3-49a1-a6e0-acb6937eecde-kube-api-access-5nz86\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:31:42.633787 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:42.633749    5689 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c6e7e1d-dfc3-49a1-a6e0-acb6937eecde-plugins-config-volume\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:31:43.204050 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:43.204012    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:31:43.389443 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:43.389390    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:31:43.420884 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:43.420862    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-9db8tm77q"
Nov 01 05:31:43.421051 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:43.420902    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-9db8tm77q" event={"ID":"8c6e7e1d-dfc3-49a1-a6e0-acb6937eecde","Type":"ContainerDied","Data":"af723480ece1fbd17e4523164267c0b9c4f7820d132ab4f80a68763c73dcabc2"}
Nov 01 05:31:43.421051 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:43.420961    5689 scope.go:122] "RemoveContainer" containerID="f24976884609d62e9ce0ae6bd4660364b3140fcb062085e4a84e0debb5af47b9"
Nov 01 05:31:43.569737 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:43.569657    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:31:43.586408 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:43.586382    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-9db8tm77q"]
Nov 01 05:31:43.636106 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:43.636082    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-9db8tm77q"]
Nov 01 05:31:44.073215 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:44.073182    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c6e7e1d-dfc3-49a1-a6e0-acb6937eecde" path="/var/lib/kubelet/pods/8c6e7e1d-dfc3-49a1-a6e0-acb6937eecde/volumes"
Nov 01 05:31:44.445332 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:44.445298    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:31:45.376133 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:45.376102    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:31:46.434855 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:46.434825    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:31:46.642077 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:46.642045    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:31:46.846521 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:46.846415    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-afc555-86567dff5-4gjlw"]
Nov 01 05:31:46.846960 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:46.846936    5689 cpu_manager.go:414] "containerMap: removing container" podUID="6775cee0-3876-46a8-b8f8-53b420c8277e" containerName="payload-processing"
Nov 01 05:31:46.846960 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:46.846956    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6775cee0-3876-46a8-b8f8-53b420c8277e" containerName="payload-processing"
Nov 01 05:31:46.847116 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:46.846984    5689 cpu_manager.go:414] "containerMap: removing container" podUID="54adae87-71b3-4dfa-8eb0-3ad86c47e5a8" containerName="istio-proxy"
Nov 01 05:31:46.847116 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:46.846992    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54adae87-71b3-4dfa-8eb0-3ad86c47e5a8" containerName="istio-proxy"
Nov 01 05:31:46.847116 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:46.847010    5689 cpu_manager.go:414] "containerMap: removing container" podUID="b68feecd-350f-4582-8cca-c966f065e929" containerName="istio-proxy"
Nov 01 05:31:46.847116 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:46.847019    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b68feecd-350f-4582-8cca-c966f065e929" containerName="istio-proxy"
Nov 01 05:31:46.847116 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:46.847040    5689 cpu_manager.go:414] "containerMap: removing container" podUID="ad96da25-2a2b-471c-9a7c-2c3f166553de" containerName="istio-proxy"
Nov 01 05:31:46.847116 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:46.847047    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad96da25-2a2b-471c-9a7c-2c3f166553de" containerName="istio-proxy"
Nov 01 05:31:46.847116 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:46.847071    5689 cpu_manager.go:414] "containerMap: removing container" podUID="494e7554-111b-4033-8a49-68ae0ad86c0b" containerName="payload-processing"
Nov 01 05:31:46.847116 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:46.847078    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="494e7554-111b-4033-8a49-68ae0ad86c0b" containerName="payload-processing"
Nov 01 05:31:46.847116 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:46.847100    5689 cpu_manager.go:414] "containerMap: removing container" podUID="880e86d4-6364-4e6d-9b59-103b1a8668aa" containerName="cleanup"
Nov 01 05:31:46.847116 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:46.847108    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="880e86d4-6364-4e6d-9b59-103b1a8668aa" containerName="cleanup"
Nov 01 05:31:46.847434 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:46.847122    5689 cpu_manager.go:414] "containerMap: removing container" podUID="8c6e7e1d-dfc3-49a1-a6e0-acb6937eecde" containerName="payload-processing"
Nov 01 05:31:46.847434 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:46.847127    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c6e7e1d-dfc3-49a1-a6e0-acb6937eecde" containerName="payload-processing"
Nov 01 05:31:46.847434 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:46.847226    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c6e7e1d-dfc3-49a1-a6e0-acb6937eecde" containerName="payload-processing"
Nov 01 05:31:46.847434 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:46.847235    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="6775cee0-3876-46a8-b8f8-53b420c8277e" containerName="payload-processing"
Nov 01 05:31:46.847434 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:46.847243    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="54adae87-71b3-4dfa-8eb0-3ad86c47e5a8" containerName="istio-proxy"
Nov 01 05:31:46.847434 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:46.847251    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="b68feecd-350f-4582-8cca-c966f065e929" containerName="istio-proxy"
Nov 01 05:31:46.847434 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:46.847258    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad96da25-2a2b-471c-9a7c-2c3f166553de" containerName="istio-proxy"
Nov 01 05:31:46.847434 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:46.847266    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="494e7554-111b-4033-8a49-68ae0ad86c0b" containerName="payload-processing"
Nov 01 05:31:46.847434 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:46.847273    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="880e86d4-6364-4e6d-9b59-103b1a8668aa" containerName="cleanup"
Nov 01 05:31:46.850790 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:46.850769    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-afc555-86567dff5-4gjlw"
Nov 01 05:31:46.892270 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:46.892237    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ae03a27-0c9c-4ca2-89c1-678ea84e7d9d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-afc555-86567dff5-4gjlw\" (UID: \"1ae03a27-0c9c-4ca2-89c1-678ea84e7d9d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-afc555-86567dff5-4gjlw"
Nov 01 05:31:46.892376 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:46.892351    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h8pq\" (UniqueName: \"kubernetes.io/projected/1ae03a27-0c9c-4ca2-89c1-678ea84e7d9d-kube-api-access-5h8pq\") pod \"payload-pre-processing-e2e-shared-a-w0-afc555-86567dff5-4gjlw\" (UID: \"1ae03a27-0c9c-4ca2-89c1-678ea84e7d9d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-afc555-86567dff5-4gjlw"
Nov 01 05:31:46.902706 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:46.902689    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-afc555\""
Nov 01 05:31:46.902942 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:46.902925    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-afc555-dockercfg-qjc8s\""
Nov 01 05:31:46.909343 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:46.909318    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-afc555-86567dff5-4gjlw"]
Nov 01 05:31:46.978540 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:46.978505    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-afc555-84db8c5b8c-kh4bq"]
Nov 01 05:31:47.030167 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:47.014513    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h8pq\" (UniqueName: \"kubernetes.io/projected/1ae03a27-0c9c-4ca2-89c1-678ea84e7d9d-kube-api-access-5h8pq\") pod \"payload-pre-processing-e2e-shared-a-w0-afc555-86567dff5-4gjlw\" (UID: \"1ae03a27-0c9c-4ca2-89c1-678ea84e7d9d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-afc555-86567dff5-4gjlw"
Nov 01 05:31:47.041550 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:47.041504    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ae03a27-0c9c-4ca2-89c1-678ea84e7d9d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-afc555-86567dff5-4gjlw\" (UID: \"1ae03a27-0c9c-4ca2-89c1-678ea84e7d9d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-afc555-86567dff5-4gjlw"
Nov 01 05:31:47.041872 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:47.041243    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-afc555-84db8c5b8c-kh4bq"]
Nov 01 05:31:47.042024 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:47.032799    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-afc555-84db8c5b8c-kh4bq"
Nov 01 05:31:47.042375 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:47.042353    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ae03a27-0c9c-4ca2-89c1-678ea84e7d9d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-afc555-86567dff5-4gjlw\" (UID: \"1ae03a27-0c9c-4ca2-89c1-678ea84e7d9d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-afc555-86567dff5-4gjlw"
Nov 01 05:31:47.080025 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:47.079995    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h8pq\" (UniqueName: \"kubernetes.io/projected/1ae03a27-0c9c-4ca2-89c1-678ea84e7d9d-kube-api-access-5h8pq\") pod \"payload-pre-processing-e2e-shared-a-w0-afc555-86567dff5-4gjlw\" (UID: \"1ae03a27-0c9c-4ca2-89c1-678ea84e7d9d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-afc555-86567dff5-4gjlw"
Nov 01 05:31:47.143018 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:47.142943    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28a2cf8f-22d0-426e-af53-a5c0128e8d5d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-afc555-84db8c5b8c-kh4bq\" (UID: \"28a2cf8f-22d0-426e-af53-a5c0128e8d5d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-afc555-84db8c5b8c-kh4bq"
Nov 01 05:31:47.143181 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:47.143135    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2zk8\" (UniqueName: \"kubernetes.io/projected/28a2cf8f-22d0-426e-af53-a5c0128e8d5d-kube-api-access-m2zk8\") pod \"payload-processing-e2e-shared-a-w0-afc555-84db8c5b8c-kh4bq\" (UID: \"28a2cf8f-22d0-426e-af53-a5c0128e8d5d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-afc555-84db8c5b8c-kh4bq"
Nov 01 05:31:47.164946 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:47.164918    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-afc555-86567dff5-4gjlw"
Nov 01 05:31:47.244708 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:47.244678    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2zk8\" (UniqueName: \"kubernetes.io/projected/28a2cf8f-22d0-426e-af53-a5c0128e8d5d-kube-api-access-m2zk8\") pod \"payload-processing-e2e-shared-a-w0-afc555-84db8c5b8c-kh4bq\" (UID: \"28a2cf8f-22d0-426e-af53-a5c0128e8d5d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-afc555-84db8c5b8c-kh4bq"
Nov 01 05:31:47.244864 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:47.244779    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28a2cf8f-22d0-426e-af53-a5c0128e8d5d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-afc555-84db8c5b8c-kh4bq\" (UID: \"28a2cf8f-22d0-426e-af53-a5c0128e8d5d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-afc555-84db8c5b8c-kh4bq"
Nov 01 05:31:47.245429 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:47.245408    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28a2cf8f-22d0-426e-af53-a5c0128e8d5d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-afc555-84db8c5b8c-kh4bq\" (UID: \"28a2cf8f-22d0-426e-af53-a5c0128e8d5d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-afc555-84db8c5b8c-kh4bq"
Nov 01 05:31:47.305119 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:47.305093    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2zk8\" (UniqueName: \"kubernetes.io/projected/28a2cf8f-22d0-426e-af53-a5c0128e8d5d-kube-api-access-m2zk8\") pod \"payload-processing-e2e-shared-a-w0-afc555-84db8c5b8c-kh4bq\" (UID: \"28a2cf8f-22d0-426e-af53-a5c0128e8d5d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-afc555-84db8c5b8c-kh4bq"
Nov 01 05:31:47.368203 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:47.368176    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-afc555-84db8c5b8c-kh4bq"
Nov 01 05:31:47.546298 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:47.546272    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-afc555-86567dff5-4gjlw"]
Nov 01 05:31:47.548966 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:31:47.548938    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ae03a27_0c9c_4ca2_89c1_678ea84e7d9d.slice/crio-893900e7bcc1234775edefebe9164720b0bc810c8966cc15146955429c41f5d9 WatchSource:0}: Error finding container 893900e7bcc1234775edefebe9164720b0bc810c8966cc15146955429c41f5d9: Status 404 returned error can't find the container with id 893900e7bcc1234775edefebe9164720b0bc810c8966cc15146955429c41f5d9
Nov 01 05:31:47.749029 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:47.749005    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-afc555-84db8c5b8c-kh4bq"]
Nov 01 05:31:47.751218 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:31:47.751189    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28a2cf8f_22d0_426e_af53_a5c0128e8d5d.slice/crio-d38323f33d9a0b26ff3bf263347003f4aced9e82a01a5658c46d1b32c471709e WatchSource:0}: Error finding container d38323f33d9a0b26ff3bf263347003f4aced9e82a01a5658c46d1b32c471709e: Status 404 returned error can't find the container with id d38323f33d9a0b26ff3bf263347003f4aced9e82a01a5658c46d1b32c471709e
Nov 01 05:31:48.454495 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:48.454448    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-afc555-86567dff5-4gjlw" event={"ID":"1ae03a27-0c9c-4ca2-89c1-678ea84e7d9d","Type":"ContainerStarted","Data":"faed5c03926ad4f9cfc8a8b6ce646c589802973b1a203bc7269056fc8c113dbb"}
Nov 01 05:31:48.454694 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:48.454516    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-afc555-86567dff5-4gjlw" event={"ID":"1ae03a27-0c9c-4ca2-89c1-678ea84e7d9d","Type":"ContainerStarted","Data":"893900e7bcc1234775edefebe9164720b0bc810c8966cc15146955429c41f5d9"}
Nov 01 05:31:48.454694 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:48.454545    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-afc555-86567dff5-4gjlw"
Nov 01 05:31:48.456139 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:48.456112    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-afc555-84db8c5b8c-kh4bq" event={"ID":"28a2cf8f-22d0-426e-af53-a5c0128e8d5d","Type":"ContainerStarted","Data":"f12935a2f8cc1a4c6fdeb66440cf38361ffac80ff3c227b6535a52179f4e3d08"}
Nov 01 05:31:48.456259 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:48.456163    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-afc555-84db8c5b8c-kh4bq" event={"ID":"28a2cf8f-22d0-426e-af53-a5c0128e8d5d","Type":"ContainerStarted","Data":"d38323f33d9a0b26ff3bf263347003f4aced9e82a01a5658c46d1b32c471709e"}
Nov 01 05:31:48.456259 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:48.456238    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-afc555-84db8c5b8c-kh4bq"
Nov 01 05:31:48.593902 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:48.593853    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-afc555-86567dff5-4gjlw" podStartSLOduration=2.593840146 podStartE2EDuration="2.593840146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:31:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:31:48.59174866 +0000 UTC m=+1686.662545391" watchObservedRunningTime="2026-11-01 05:31:48.593840146 +0000 UTC m=+1686.664636891"
Nov 01 05:31:48.741336 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:48.741193    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-afc555-84db8c5b8c-kh4bq" podStartSLOduration=2.741173895 podStartE2EDuration="2.741173895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:31:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:31:48.740633089 +0000 UTC m=+1686.811429811" watchObservedRunningTime="2026-11-01 05:31:48.741173895 +0000 UTC m=+1686.811970616"
Nov 01 05:31:50.352225 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:50.352193    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:31:50.535258 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:50.535222    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:31:51.137054 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:51.137020    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:31:51.263423 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:51.263396    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ca7a55-74bd57dbd9-jrwrq"]
Nov 01 05:31:51.372597 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:51.372561    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ca7a55-747b56569f-5s7gh"]
Nov 01 05:31:51.669046 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:51.669013    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:31:51.843494 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:51.843418    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:31:51.911478 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:51.907831    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4"]
Nov 01 05:31:51.946186 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:51.946112    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4"
Nov 01 05:31:51.967094 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:51.967061    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4"]
Nov 01 05:31:52.007680 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:52.007643    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d885f72a-2f29-41ee-b803-ab1a3a96fc32-credential-socket\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4\" (UID: \"d885f72a-2f29-41ee-b803-ab1a3a96fc32\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4"
Nov 01 05:31:52.007978 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:52.007955    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d885f72a-2f29-41ee-b803-ab1a3a96fc32-istio-token\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4\" (UID: \"d885f72a-2f29-41ee-b803-ab1a3a96fc32\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4"
Nov 01 05:31:52.008126 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:52.008109    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d885f72a-2f29-41ee-b803-ab1a3a96fc32-istio-podinfo\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4\" (UID: \"d885f72a-2f29-41ee-b803-ab1a3a96fc32\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4"
Nov 01 05:31:52.008343 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:52.008327    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d885f72a-2f29-41ee-b803-ab1a3a96fc32-istiod-ca-cert\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4\" (UID: \"d885f72a-2f29-41ee-b803-ab1a3a96fc32\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4"
Nov 01 05:31:52.008536 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:52.008516    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d885f72a-2f29-41ee-b803-ab1a3a96fc32-istio-envoy\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4\" (UID: \"d885f72a-2f29-41ee-b803-ab1a3a96fc32\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4"
Nov 01 05:31:52.008699 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:52.008684    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d885f72a-2f29-41ee-b803-ab1a3a96fc32-workload-certs\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4\" (UID: \"d885f72a-2f29-41ee-b803-ab1a3a96fc32\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4"
Nov 01 05:31:52.008850 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:52.008835    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d885f72a-2f29-41ee-b803-ab1a3a96fc32-istio-data\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4\" (UID: \"d885f72a-2f29-41ee-b803-ab1a3a96fc32\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4"
Nov 01 05:31:52.008989 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:52.008975    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d885f72a-2f29-41ee-b803-ab1a3a96fc32-workload-socket\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4\" (UID: \"d885f72a-2f29-41ee-b803-ab1a3a96fc32\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4"
Nov 01 05:31:52.009139 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:52.009126    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv7pc\" (UniqueName: \"kubernetes.io/projected/d885f72a-2f29-41ee-b803-ab1a3a96fc32-kube-api-access-jv7pc\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4\" (UID: \"d885f72a-2f29-41ee-b803-ab1a3a96fc32\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4"
Nov 01 05:31:52.110774 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:52.110743    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d885f72a-2f29-41ee-b803-ab1a3a96fc32-credential-socket\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4\" (UID: \"d885f72a-2f29-41ee-b803-ab1a3a96fc32\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4"
Nov 01 05:31:52.111002 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:52.110983    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d885f72a-2f29-41ee-b803-ab1a3a96fc32-istio-token\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4\" (UID: \"d885f72a-2f29-41ee-b803-ab1a3a96fc32\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4"
Nov 01 05:31:52.111091 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:52.111020    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d885f72a-2f29-41ee-b803-ab1a3a96fc32-istio-podinfo\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4\" (UID: \"d885f72a-2f29-41ee-b803-ab1a3a96fc32\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4"
Nov 01 05:31:52.111151 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:52.111123    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d885f72a-2f29-41ee-b803-ab1a3a96fc32-istiod-ca-cert\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4\" (UID: \"d885f72a-2f29-41ee-b803-ab1a3a96fc32\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4"
Nov 01 05:31:52.111207 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:52.111158    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d885f72a-2f29-41ee-b803-ab1a3a96fc32-istio-envoy\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4\" (UID: \"d885f72a-2f29-41ee-b803-ab1a3a96fc32\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4"
Nov 01 05:31:52.111207 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:52.111195    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d885f72a-2f29-41ee-b803-ab1a3a96fc32-workload-certs\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4\" (UID: \"d885f72a-2f29-41ee-b803-ab1a3a96fc32\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4"
Nov 01 05:31:52.111336 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:52.111214    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d885f72a-2f29-41ee-b803-ab1a3a96fc32-credential-socket\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4\" (UID: \"d885f72a-2f29-41ee-b803-ab1a3a96fc32\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4"
Nov 01 05:31:52.111515 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:52.111491    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d885f72a-2f29-41ee-b803-ab1a3a96fc32-workload-certs\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4\" (UID: \"d885f72a-2f29-41ee-b803-ab1a3a96fc32\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4"
Nov 01 05:31:52.111699 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:52.111569    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d885f72a-2f29-41ee-b803-ab1a3a96fc32-istio-data\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4\" (UID: \"d885f72a-2f29-41ee-b803-ab1a3a96fc32\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4"
Nov 01 05:31:52.111834 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:52.111808    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d885f72a-2f29-41ee-b803-ab1a3a96fc32-istio-data\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4\" (UID: \"d885f72a-2f29-41ee-b803-ab1a3a96fc32\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4"
Nov 01 05:31:52.111960 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:52.111941    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d885f72a-2f29-41ee-b803-ab1a3a96fc32-workload-socket\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4\" (UID: \"d885f72a-2f29-41ee-b803-ab1a3a96fc32\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4"
Nov 01 05:31:52.112441 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:52.112422    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv7pc\" (UniqueName: \"kubernetes.io/projected/d885f72a-2f29-41ee-b803-ab1a3a96fc32-kube-api-access-jv7pc\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4\" (UID: \"d885f72a-2f29-41ee-b803-ab1a3a96fc32\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4"
Nov 01 05:31:52.112733 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:52.112708    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d885f72a-2f29-41ee-b803-ab1a3a96fc32-istiod-ca-cert\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4\" (UID: \"d885f72a-2f29-41ee-b803-ab1a3a96fc32\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4"
Nov 01 05:31:52.112818 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:52.112347    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d885f72a-2f29-41ee-b803-ab1a3a96fc32-workload-socket\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4\" (UID: \"d885f72a-2f29-41ee-b803-ab1a3a96fc32\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4"
Nov 01 05:31:52.114834 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:52.114755    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d885f72a-2f29-41ee-b803-ab1a3a96fc32-istio-envoy\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4\" (UID: \"d885f72a-2f29-41ee-b803-ab1a3a96fc32\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4"
Nov 01 05:31:52.116023 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:52.116000    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d885f72a-2f29-41ee-b803-ab1a3a96fc32-istio-podinfo\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4\" (UID: \"d885f72a-2f29-41ee-b803-ab1a3a96fc32\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4"
Nov 01 05:31:52.171838 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:52.171806    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d885f72a-2f29-41ee-b803-ab1a3a96fc32-istio-token\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4\" (UID: \"d885f72a-2f29-41ee-b803-ab1a3a96fc32\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4"
Nov 01 05:31:52.173028 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:52.173008    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv7pc\" (UniqueName: \"kubernetes.io/projected/d885f72a-2f29-41ee-b803-ab1a3a96fc32-kube-api-access-jv7pc\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4\" (UID: \"d885f72a-2f29-41ee-b803-ab1a3a96fc32\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4"
Nov 01 05:31:52.283241 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:52.283128    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4"
Nov 01 05:31:52.792481 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:52.789831    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4"]
Nov 01 05:31:52.810641 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:52.807482    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:31:52.810641 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:52.807640    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:31:52.810641 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:52.807733    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:31:53.495487 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:53.495370    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4" event={"ID":"d885f72a-2f29-41ee-b803-ab1a3a96fc32","Type":"ContainerStarted","Data":"daec85060065e22ccd8f7539888b7f6ec2d75b876fa53258eb3164ff68a3fb6e"}
Nov 01 05:31:53.495487 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:53.495439    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4" event={"ID":"d885f72a-2f29-41ee-b803-ab1a3a96fc32","Type":"ContainerStarted","Data":"5381bed14b6c7e57579e331ccaa76690f38504f1f4343e2828bcba5273f30b46"}
Nov 01 05:31:53.673191 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:53.673133    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4" podStartSLOduration=2.673118595 podStartE2EDuration="2.673118595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:31:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:31:53.671033338 +0000 UTC m=+1691.741830058" watchObservedRunningTime="2026-11-01 05:31:53.673118595 +0000 UTC m=+1691.743915408"
Nov 01 05:31:54.293584 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:54.288563    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4"
Nov 01 05:31:54.293584 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:54.288731    5689 patch_prober.go:29] interesting pod/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Nov 01 05:31:54.293584 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:54.288782    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4" podUID="d885f72a-2f29-41ee-b803-ab1a3a96fc32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Nov 01 05:31:55.292478 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:55.288782    5689 patch_prober.go:29] interesting pod/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Nov 01 05:31:55.292478 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:55.288845    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4" podUID="d885f72a-2f29-41ee-b803-ab1a3a96fc32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Nov 01 05:31:55.431214 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:55.431169    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:31:56.264288 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:56.264228    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-ca7a55-74bd57dbd9-jrwrq" podUID="a984df19-49d0-4c90-a51d-b1028220ee61" containerName="payload-processing" containerID="cri-o://892e4c3e67a46b02f44c8fdaa1eb14f6e036ca66fa77559ab159deeaa52165ac" gracePeriod=25
Nov 01 05:31:56.285815 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:56.285783    5689 patch_prober.go:29] interesting pod/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Nov 01 05:31:56.286014 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:56.285987    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4" podUID="d885f72a-2f29-41ee-b803-ab1a3a96fc32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Nov 01 05:31:56.299294 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:56.299226    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:31:56.374111 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:56.374074    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ca7a55-747b56569f-5s7gh" podUID="331a126d-11b4-4e43-88c3-5f2459793348" containerName="payload-pre-processing" containerID="cri-o://16a12207a91113b992bb00d61786a7e70a0794347be0ca8af6b7f6dd7ae36a32" gracePeriod=25
Nov 01 05:31:56.525258 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:56.525226    5689 generic.go:356] "Generic (PLEG): container finished" podID="331a126d-11b4-4e43-88c3-5f2459793348" containerID="16a12207a91113b992bb00d61786a7e70a0794347be0ca8af6b7f6dd7ae36a32" exitCode=0
Nov 01 05:31:56.525685 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:56.525322    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ca7a55-747b56569f-5s7gh" event={"ID":"331a126d-11b4-4e43-88c3-5f2459793348","Type":"ContainerDied","Data":"16a12207a91113b992bb00d61786a7e70a0794347be0ca8af6b7f6dd7ae36a32"}
Nov 01 05:31:56.527384 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:56.527360    5689 generic.go:356] "Generic (PLEG): container finished" podID="a984df19-49d0-4c90-a51d-b1028220ee61" containerID="892e4c3e67a46b02f44c8fdaa1eb14f6e036ca66fa77559ab159deeaa52165ac" exitCode=0
Nov 01 05:31:56.527572 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:56.527535    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-ca7a55-74bd57dbd9-jrwrq" event={"ID":"a984df19-49d0-4c90-a51d-b1028220ee61","Type":"ContainerDied","Data":"892e4c3e67a46b02f44c8fdaa1eb14f6e036ca66fa77559ab159deeaa52165ac"}
Nov 01 05:31:56.608911 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:56.608872    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-ca7a55-74bd57dbd9-jrwrq"
Nov 01 05:31:56.722149 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:56.722127    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ca7a55-747b56569f-5s7gh"
Nov 01 05:31:56.728241 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:56.728208    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a984df19-49d0-4c90-a51d-b1028220ee61-kube-api-access-t6wbq\" (UniqueName: \"kubernetes.io/projected/a984df19-49d0-4c90-a51d-b1028220ee61-kube-api-access-t6wbq\") pod \"a984df19-49d0-4c90-a51d-b1028220ee61\" (UID: \"a984df19-49d0-4c90-a51d-b1028220ee61\") "
Nov 01 05:31:56.728379 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:56.728364    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a984df19-49d0-4c90-a51d-b1028220ee61-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a984df19-49d0-4c90-a51d-b1028220ee61-plugins-config-volume\") pod \"a984df19-49d0-4c90-a51d-b1028220ee61\" (UID: \"a984df19-49d0-4c90-a51d-b1028220ee61\") "
Nov 01 05:31:56.728817 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:56.728786    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a984df19-49d0-4c90-a51d-b1028220ee61-plugins-config-volume" pod "a984df19-49d0-4c90-a51d-b1028220ee61" (UID: "a984df19-49d0-4c90-a51d-b1028220ee61"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:31:56.729120 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:56.729099    5689 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a984df19-49d0-4c90-a51d-b1028220ee61-plugins-config-volume\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:31:56.730708 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:56.730684    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a984df19-49d0-4c90-a51d-b1028220ee61-kube-api-access-t6wbq" pod "a984df19-49d0-4c90-a51d-b1028220ee61" (UID: "a984df19-49d0-4c90-a51d-b1028220ee61"). InnerVolumeSpecName "kube-api-access-t6wbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:31:56.830084 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:56.829997    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/331a126d-11b4-4e43-88c3-5f2459793348-kube-api-access-gwxwz\" (UniqueName: \"kubernetes.io/projected/331a126d-11b4-4e43-88c3-5f2459793348-kube-api-access-gwxwz\") pod \"331a126d-11b4-4e43-88c3-5f2459793348\" (UID: \"331a126d-11b4-4e43-88c3-5f2459793348\") "
Nov 01 05:31:56.830262 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:56.830198    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/331a126d-11b4-4e43-88c3-5f2459793348-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/331a126d-11b4-4e43-88c3-5f2459793348-plugins-config-volume\") pod \"331a126d-11b4-4e43-88c3-5f2459793348\" (UID: \"331a126d-11b4-4e43-88c3-5f2459793348\") "
Nov 01 05:31:56.830540 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:56.830523    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6wbq\" (UniqueName: \"kubernetes.io/projected/a984df19-49d0-4c90-a51d-b1028220ee61-kube-api-access-t6wbq\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:31:56.830626 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:56.830573    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/331a126d-11b4-4e43-88c3-5f2459793348-plugins-config-volume" pod "331a126d-11b4-4e43-88c3-5f2459793348" (UID: "331a126d-11b4-4e43-88c3-5f2459793348"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:31:56.832236 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:56.832218    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/331a126d-11b4-4e43-88c3-5f2459793348-kube-api-access-gwxwz" pod "331a126d-11b4-4e43-88c3-5f2459793348" (UID: "331a126d-11b4-4e43-88c3-5f2459793348"). InnerVolumeSpecName "kube-api-access-gwxwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:31:56.932137 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:56.932100    5689 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/331a126d-11b4-4e43-88c3-5f2459793348-plugins-config-volume\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:31:56.932137 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:56.932130    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwxwz\" (UniqueName: \"kubernetes.io/projected/331a126d-11b4-4e43-88c3-5f2459793348-kube-api-access-gwxwz\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:31:57.298476 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:57.288004    5689 patch_prober.go:29] interesting pod/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Nov 01 05:31:57.298476 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:57.288075    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4" podUID="d885f72a-2f29-41ee-b803-ab1a3a96fc32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Nov 01 05:31:57.534828 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:57.534799    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ca7a55-747b56569f-5s7gh"
Nov 01 05:31:57.535322 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:57.534806    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ca7a55-747b56569f-5s7gh" event={"ID":"331a126d-11b4-4e43-88c3-5f2459793348","Type":"ContainerDied","Data":"8a6242e9837d137f204fb0760391038bc3619dae9ab406fd5d861bdc26a819b3"}
Nov 01 05:31:57.535322 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:57.534970    5689 scope.go:122] "RemoveContainer" containerID="16a12207a91113b992bb00d61786a7e70a0794347be0ca8af6b7f6dd7ae36a32"
Nov 01 05:31:57.536685 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:57.536476    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-ca7a55-74bd57dbd9-jrwrq" event={"ID":"a984df19-49d0-4c90-a51d-b1028220ee61","Type":"ContainerDied","Data":"36e9eec07f1489d6fa6d61b955f3a83024970cbbe973064472483a5ac943b581"}
Nov 01 05:31:57.536685 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:57.536566    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-ca7a55-74bd57dbd9-jrwrq"
Nov 01 05:31:57.545988 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:57.545910    5689 scope.go:122] "RemoveContainer" containerID="892e4c3e67a46b02f44c8fdaa1eb14f6e036ca66fa77559ab159deeaa52165ac"
Nov 01 05:31:57.706589 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:57.706545    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ca7a55-747b56569f-5s7gh"]
Nov 01 05:31:57.755842 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:57.755812    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ca7a55-747b56569f-5s7gh"]
Nov 01 05:31:57.822790 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:57.822709    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ca7a55-74bd57dbd9-jrwrq"]
Nov 01 05:31:57.827019 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:57.826988    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ca7a55-74bd57dbd9-jrwrq"]
Nov 01 05:31:58.067545 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:58.067518    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="331a126d-11b4-4e43-88c3-5f2459793348" path="/var/lib/kubelet/pods/331a126d-11b4-4e43-88c3-5f2459793348/volumes"
Nov 01 05:31:58.067899 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:58.067885    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a984df19-49d0-4c90-a51d-b1028220ee61" path="/var/lib/kubelet/pods/a984df19-49d0-4c90-a51d-b1028220ee61/volumes"
Nov 01 05:31:58.292484 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:58.291828    5689 patch_prober.go:29] interesting pod/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Nov 01 05:31:58.292484 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:58.291902    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4" podUID="d885f72a-2f29-41ee-b803-ab1a3a96fc32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Nov 01 05:31:59.284436 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:59.284392    5689 patch_prober.go:29] interesting pod/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Nov 01 05:31:59.284860 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:59.284491    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4" podUID="d885f72a-2f29-41ee-b803-ab1a3a96fc32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Nov 01 05:31:59.467179 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:59.467144    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-afc555-86567dff5-4gjlw"
Nov 01 05:31:59.467528 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:31:59.467508    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-afc555-84db8c5b8c-kh4bq"
Nov 01 05:32:00.070393 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:32:00.070356    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:32:00.071820 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:32:00.071790    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:32:00.305576 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:32:00.305541    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:32:00.492148 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:32:00.492112    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4"
Nov 01 05:32:00.494018 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:32:00.493391    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4"
Nov 01 05:32:00.495130 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:32:00.495109    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4"
Nov 01 05:32:03.260128 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:32:03.260099    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:32:03.438205 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:32:03.438123    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:32:05.163121 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:32:05.158962    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:32:10.016594 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:32:10.016562    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:32:14.816219 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:32:14.816181    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:32:19.625249 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:32:19.625216    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:32:23.800994 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:32:23.800906    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:32:25.228979 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:32:25.228945    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-afc555-d4c6bcc8f-fnlj6"]
Nov 01 05:32:25.229517 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:32:25.229493    5689 cpu_manager.go:414] "containerMap: removing container" podUID="a984df19-49d0-4c90-a51d-b1028220ee61" containerName="payload-processing"
Nov 01 05:32:25.229517 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:32:25.229517    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a984df19-49d0-4c90-a51d-b1028220ee61" containerName="payload-processing"
Nov 01 05:32:25.229614 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:32:25.229556    5689 cpu_manager.go:414] "containerMap: removing container" podUID="331a126d-11b4-4e43-88c3-5f2459793348" containerName="payload-pre-processing"
Nov 01 05:32:25.229614 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:32:25.229561    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="331a126d-11b4-4e43-88c3-5f2459793348" containerName="payload-pre-processing"
Nov 01 05:32:25.229677 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:32:25.229635    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="331a126d-11b4-4e43-88c3-5f2459793348" containerName="payload-pre-processing"
Nov 01 05:32:25.229677 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:32:25.229647    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="a984df19-49d0-4c90-a51d-b1028220ee61" containerName="payload-processing"
Nov 01 05:32:25.233906 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:32:25.233890    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-afc555-d4c6bcc8f-fnlj6"
Nov 01 05:32:25.285740 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:32:25.285720    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-afc555-serving-cert\""
Nov 01 05:32:25.314769 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:32:25.314739    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btx6p\" (UniqueName: \"kubernetes.io/projected/0226e80d-bc64-44a1-86cf-cde37ac9277c-kube-api-access-btx6p\") pod \"maas-api-e2e-shared-a-w0-afc555-d4c6bcc8f-fnlj6\" (UID: \"0226e80d-bc64-44a1-86cf-cde37ac9277c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-afc555-d4c6bcc8f-fnlj6"
Nov 01 05:32:25.314901 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:32:25.314879    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0226e80d-bc64-44a1-86cf-cde37ac9277c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-afc555-d4c6bcc8f-fnlj6\" (UID: \"0226e80d-bc64-44a1-86cf-cde37ac9277c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-afc555-d4c6bcc8f-fnlj6"
Nov 01 05:32:25.314942 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:32:25.314911    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0226e80d-bc64-44a1-86cf-cde37ac9277c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-afc555-d4c6bcc8f-fnlj6\" (UID: \"0226e80d-bc64-44a1-86cf-cde37ac9277c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-afc555-d4c6bcc8f-fnlj6"
Nov 01 05:32:25.314942 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:32:25.314928    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0226e80d-bc64-44a1-86cf-cde37ac9277c-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-afc555-d4c6bcc8f-fnlj6\" (UID: \"0226e80d-bc64-44a1-86cf-cde37ac9277c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-afc555-d4c6bcc8f-fnlj6"
Nov 01 05:32:25.340512 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:32:25.340483    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-afc555-d4c6bcc8f-fnlj6"]
Nov 01 05:32:25.415833 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:32:25.415799    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0226e80d-bc64-44a1-86cf-cde37ac9277c-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-afc555-d4c6bcc8f-fnlj6\" (UID: \"0226e80d-bc64-44a1-86cf-cde37ac9277c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-afc555-d4c6bcc8f-fnlj6"
Nov 01 05:32:25.416005 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:32:25.415942    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btx6p\" (UniqueName: \"kubernetes.io/projected/0226e80d-bc64-44a1-86cf-cde37ac9277c-kube-api-access-btx6p\") pod \"maas-api-e2e-shared-a-w0-afc555-d4c6bcc8f-fnlj6\" (UID: \"0226e80d-bc64-44a1-86cf-cde37ac9277c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-afc555-d4c6bcc8f-fnlj6"
Nov 01 05:32:25.416099 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:32:25.416085    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0226e80d-bc64-44a1-86cf-cde37ac9277c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-afc555-d4c6bcc8f-fnlj6\" (UID: \"0226e80d-bc64-44a1-86cf-cde37ac9277c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-afc555-d4c6bcc8f-fnlj6"
Nov 01 05:32:25.416167 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:32:25.416130    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0226e80d-bc64-44a1-86cf-cde37ac9277c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-afc555-d4c6bcc8f-fnlj6\" (UID: \"0226e80d-bc64-44a1-86cf-cde37ac9277c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-afc555-d4c6bcc8f-fnlj6"
Nov 01 05:32:25.416260 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:32:25.416245    5689 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-afc555-serving-cert: secret "maas-api-e2e-shared-a-w0-afc555-serving-cert" not found
Nov 01 05:32:25.416340 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:32:25.416329    5689 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0226e80d-bc64-44a1-86cf-cde37ac9277c-maas-api-tls podName:0226e80d-bc64-44a1-86cf-cde37ac9277c nodeName:}" failed. No retries permitted until 2026-11-01 05:32:25.916307534 +0000 UTC m=+1723.987104233 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0226e80d-bc64-44a1-86cf-cde37ac9277c-maas-api-tls") pod "maas-api-e2e-shared-a-w0-afc555-d4c6bcc8f-fnlj6" (UID: "0226e80d-bc64-44a1-86cf-cde37ac9277c") : secret "maas-api-e2e-shared-a-w0-afc555-serving-cert" not found
Nov 01 05:32:25.416697 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:32:25.416677    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0226e80d-bc64-44a1-86cf-cde37ac9277c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-afc555-d4c6bcc8f-fnlj6\" (UID: \"0226e80d-bc64-44a1-86cf-cde37ac9277c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-afc555-d4c6bcc8f-fnlj6"
Nov 01 05:32:25.418349 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:32:25.418330    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0226e80d-bc64-44a1-86cf-cde37ac9277c-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-afc555-d4c6bcc8f-fnlj6\" (UID: \"0226e80d-bc64-44a1-86cf-cde37ac9277c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-afc555-d4c6bcc8f-fnlj6"
Nov 01 05:32:25.453575 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:32:25.453542    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:32:25.477334 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:32:25.477306    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btx6p\" (UniqueName: \"kubernetes.io/projected/0226e80d-bc64-44a1-86cf-cde37ac9277c-kube-api-access-btx6p\") pod \"maas-api-e2e-shared-a-w0-afc555-d4c6bcc8f-fnlj6\" (UID: \"0226e80d-bc64-44a1-86cf-cde37ac9277c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-afc555-d4c6bcc8f-fnlj6"
Nov 01 05:32:25.646801 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:32:25.646717    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:32:25.833628 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:32:25.833594    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:32:25.920903 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:32:25.920821    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0226e80d-bc64-44a1-86cf-cde37ac9277c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-afc555-d4c6bcc8f-fnlj6\" (UID: \"0226e80d-bc64-44a1-86cf-cde37ac9277c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-afc555-d4c6bcc8f-fnlj6"
Nov 01 05:32:25.923353 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:32:25.923330    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0226e80d-bc64-44a1-86cf-cde37ac9277c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-afc555-d4c6bcc8f-fnlj6\" (UID: \"0226e80d-bc64-44a1-86cf-cde37ac9277c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-afc555-d4c6bcc8f-fnlj6"
Nov 01 05:32:26.147532 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:32:26.147500    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-afc555-d4c6bcc8f-fnlj6"
Nov 01 05:32:26.527069 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:32:26.527044    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-afc555-d4c6bcc8f-fnlj6"]
Nov 01 05:32:26.529631 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:32:26.529600    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0226e80d_bc64_44a1_86cf_cde37ac9277c.slice/crio-c782a9dbda6dbb1d16b252b693b8136497d159689d537ae847586088de98fc3e WatchSource:0}: Error finding container c782a9dbda6dbb1d16b252b693b8136497d159689d537ae847586088de98fc3e: Status 404 returned error can't find the container with id c782a9dbda6dbb1d16b252b693b8136497d159689d537ae847586088de98fc3e
Nov 01 05:32:26.739091 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:32:26.739051    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-afc555-d4c6bcc8f-fnlj6" event={"ID":"0226e80d-bc64-44a1-86cf-cde37ac9277c","Type":"ContainerStarted","Data":"c782a9dbda6dbb1d16b252b693b8136497d159689d537ae847586088de98fc3e"}
Nov 01 05:32:27.746139 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:32:27.746060    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-afc555-d4c6bcc8f-fnlj6" event={"ID":"0226e80d-bc64-44a1-86cf-cde37ac9277c","Type":"ContainerStarted","Data":"f3679e4fd5ce7d1783402079b40f384abd5d3b565d34ca21e6feb1de5b26927b"}
Nov 01 05:32:27.746525 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:32:27.746183    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-afc555-d4c6bcc8f-fnlj6"
Nov 01 05:32:27.866114 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:32:27.866053    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-afc555-d4c6bcc8f-fnlj6" podStartSLOduration=1.986552728 podStartE2EDuration="2.866039926s" totalImagesPullingTime="879.487198ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:32:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:32:27.86377963 +0000 UTC m=+1725.934576351" watchObservedRunningTime="2026-11-01 05:32:27.866039926 +0000 UTC m=+1725.936836646"
Nov 01 05:32:28.624321 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:32:28.624289    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:32:28.800288 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:32:28.800248    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:32:30.069678 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:32:30.069646    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:32:30.146394 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:32:30.146371    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:32:30.583721 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:32:30.583688    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:32:30.775543 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:32:30.775510    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:32:33.287434 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:32:33.287397    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:32:33.468868 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:32:33.468837    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:32:33.757862 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:32:33.757834    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-afc555-d4c6bcc8f-fnlj6"
Nov 01 05:32:33.998198 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:32:33.998159    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:32:35.718566 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:32:35.718516    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:32:36.963626 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:32:36.963521    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:32:37.146555 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:32:37.146519    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:32:39.331048 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:32:39.331015    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:32:40.544218 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:32:40.544151    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:32:40.730414 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:32:40.730373    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:32:40.912530 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:32:40.912503    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:32:41.533894 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:32:41.533855    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:32:42.164286 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:32:42.164254    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:32:44.859888 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:32:44.859855    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:32:45.044362 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:32:45.044314    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:32:45.552240 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:32:45.552206    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:32:48.111047 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:32:48.111014    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:32:48.289943 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:32:48.289897    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:32:50.614037 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:32:50.614003    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:32:55.522712 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:32:55.522679    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:33:00.069388 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:33:00.069353    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:33:00.314970 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:33:00.314932    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:33:03.598781 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:33:03.598704    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:33:05.127213 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:33:05.116856    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:33:09.908768 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:33:09.908740    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:33:13.289080 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:33:13.289051    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:33:15.442716 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:33:15.442684    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:33:20.037472 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:33:20.032354    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:33:20.210193 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:33:20.210156    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:33:20.547181 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:33:20.547145    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:33:21.031927 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:33:21.031890    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm"]
Nov 01 05:33:21.036163 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:33:21.036140    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm"
Nov 01 05:33:21.098758 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:33:21.098717    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q282\" (UniqueName: \"kubernetes.io/projected/b7183c3f-d319-4c01-83fe-0df57b41b69c-kube-api-access-8q282\") pod \"e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm\" (UID: \"b7183c3f-d319-4c01-83fe-0df57b41b69c\") " pod="openshift-ingress/e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm"
Nov 01 05:33:21.099219 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:33:21.098776    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7183c3f-d319-4c01-83fe-0df57b41b69c-istiod-ca-cert\") pod \"e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm\" (UID: \"b7183c3f-d319-4c01-83fe-0df57b41b69c\") " pod="openshift-ingress/e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm"
Nov 01 05:33:21.099219 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:33:21.098843    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7183c3f-d319-4c01-83fe-0df57b41b69c-workload-socket\") pod \"e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm\" (UID: \"b7183c3f-d319-4c01-83fe-0df57b41b69c\") " pod="openshift-ingress/e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm"
Nov 01 05:33:21.099219 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:33:21.098866    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7183c3f-d319-4c01-83fe-0df57b41b69c-istio-podinfo\") pod \"e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm\" (UID: \"b7183c3f-d319-4c01-83fe-0df57b41b69c\") " pod="openshift-ingress/e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm"
Nov 01 05:33:21.099219 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:33:21.098960    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7183c3f-d319-4c01-83fe-0df57b41b69c-istio-envoy\") pod \"e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm\" (UID: \"b7183c3f-d319-4c01-83fe-0df57b41b69c\") " pod="openshift-ingress/e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm"
Nov 01 05:33:21.099219 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:33:21.099021    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7183c3f-d319-4c01-83fe-0df57b41b69c-istio-data\") pod \"e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm\" (UID: \"b7183c3f-d319-4c01-83fe-0df57b41b69c\") " pod="openshift-ingress/e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm"
Nov 01 05:33:21.099219 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:33:21.099063    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7183c3f-d319-4c01-83fe-0df57b41b69c-credential-socket\") pod \"e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm\" (UID: \"b7183c3f-d319-4c01-83fe-0df57b41b69c\") " pod="openshift-ingress/e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm"
Nov 01 05:33:21.099219 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:33:21.099113    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7183c3f-d319-4c01-83fe-0df57b41b69c-workload-certs\") pod \"e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm\" (UID: \"b7183c3f-d319-4c01-83fe-0df57b41b69c\") " pod="openshift-ingress/e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm"
Nov 01 05:33:21.099219 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:33:21.099161    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7183c3f-d319-4c01-83fe-0df57b41b69c-istio-token\") pod \"e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm\" (UID: \"b7183c3f-d319-4c01-83fe-0df57b41b69c\") " pod="openshift-ingress/e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm"
Nov 01 05:33:21.108378 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:33:21.108356    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-1c4191-openshift-default-dockercfg-vngqp\""
Nov 01 05:33:21.141368 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:33:21.141343    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm"]
Nov 01 05:33:21.161149 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:33:21.161123    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:33:21.249488 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:33:21.200557    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q282\" (UniqueName: \"kubernetes.io/projected/b7183c3f-d319-4c01-83fe-0df57b41b69c-kube-api-access-8q282\") pod \"e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm\" (UID: \"b7183c3f-d319-4c01-83fe-0df57b41b69c\") " pod="openshift-ingress/e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm"
Nov 01 05:33:21.249809 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:33:21.249783    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7183c3f-d319-4c01-83fe-0df57b41b69c-istiod-ca-cert\") pod \"e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm\" (UID: \"b7183c3f-d319-4c01-83fe-0df57b41b69c\") " pod="openshift-ingress/e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm"
Nov 01 05:33:21.249986 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:33:21.249968    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7183c3f-d319-4c01-83fe-0df57b41b69c-workload-socket\") pod \"e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm\" (UID: \"b7183c3f-d319-4c01-83fe-0df57b41b69c\") " pod="openshift-ingress/e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm"
Nov 01 05:33:21.250102 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:33:21.250087    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7183c3f-d319-4c01-83fe-0df57b41b69c-istio-podinfo\") pod \"e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm\" (UID: \"b7183c3f-d319-4c01-83fe-0df57b41b69c\") " pod="openshift-ingress/e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm"
Nov 01 05:33:21.250289 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:33:21.250275    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7183c3f-d319-4c01-83fe-0df57b41b69c-istio-envoy\") pod \"e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm\" (UID: \"b7183c3f-d319-4c01-83fe-0df57b41b69c\") " pod="openshift-ingress/e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm"
Nov 01 05:33:21.273476 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:33:21.250486    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7183c3f-d319-4c01-83fe-0df57b41b69c-istio-data\") pod \"e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm\" (UID: \"b7183c3f-d319-4c01-83fe-0df57b41b69c\") " pod="openshift-ingress/e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm"
Nov 01 05:33:21.273744 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:33:21.254996    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7183c3f-d319-4c01-83fe-0df57b41b69c-workload-socket\") pod \"e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm\" (UID: \"b7183c3f-d319-4c01-83fe-0df57b41b69c\") " pod="openshift-ingress/e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm"
Nov 01 05:33:21.273836 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:33:21.255668    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7183c3f-d319-4c01-83fe-0df57b41b69c-istiod-ca-cert\") pod \"e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm\" (UID: \"b7183c3f-d319-4c01-83fe-0df57b41b69c\") " pod="openshift-ingress/e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm"
Nov 01 05:33:21.273836 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:33:21.259864    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7183c3f-d319-4c01-83fe-0df57b41b69c-istio-data\") pod \"e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm\" (UID: \"b7183c3f-d319-4c01-83fe-0df57b41b69c\") " pod="openshift-ingress/e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm"
Nov 01 05:33:21.274018 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:33:21.273994    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7183c3f-d319-4c01-83fe-0df57b41b69c-credential-socket\") pod \"e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm\" (UID: \"b7183c3f-d319-4c01-83fe-0df57b41b69c\") " pod="openshift-ingress/e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm"
Nov 01 05:33:21.291237 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:33:21.291159    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7183c3f-d319-4c01-83fe-0df57b41b69c-workload-certs\") pod \"e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm\" (UID: \"b7183c3f-d319-4c01-83fe-0df57b41b69c\") " pod="openshift-ingress/e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm"
Nov 01 05:33:21.302950 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:33:21.302919    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7183c3f-d319-4c01-83fe-0df57b41b69c-istio-token\") pod \"e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm\" (UID: \"b7183c3f-d319-4c01-83fe-0df57b41b69c\") " pod="openshift-ingress/e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm"
Nov 01 05:33:21.318403 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:33:21.275246    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7183c3f-d319-4c01-83fe-0df57b41b69c-istio-envoy\") pod \"e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm\" (UID: \"b7183c3f-d319-4c01-83fe-0df57b41b69c\") " pod="openshift-ingress/e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm"
Nov 01 05:33:21.318403 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:33:21.274511    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7183c3f-d319-4c01-83fe-0df57b41b69c-credential-socket\") pod \"e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm\" (UID: \"b7183c3f-d319-4c01-83fe-0df57b41b69c\") " pod="openshift-ingress/e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm"
Nov 01 05:33:21.318403 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:33:21.291835    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7183c3f-d319-4c01-83fe-0df57b41b69c-workload-certs\") pod \"e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm\" (UID: \"b7183c3f-d319-4c01-83fe-0df57b41b69c\") " pod="openshift-ingress/e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm"
Nov 01 05:33:21.318403 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:33:21.274992    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7183c3f-d319-4c01-83fe-0df57b41b69c-istio-podinfo\") pod \"e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm\" (UID: \"b7183c3f-d319-4c01-83fe-0df57b41b69c\") " pod="openshift-ingress/e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm"
Nov 01 05:33:21.318403 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:33:21.309441    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q282\" (UniqueName: \"kubernetes.io/projected/b7183c3f-d319-4c01-83fe-0df57b41b69c-kube-api-access-8q282\") pod \"e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm\" (UID: \"b7183c3f-d319-4c01-83fe-0df57b41b69c\") " pod="openshift-ingress/e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm"
Nov 01 05:33:21.364044 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:33:21.364015    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7183c3f-d319-4c01-83fe-0df57b41b69c-istio-token\") pod \"e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm\" (UID: \"b7183c3f-d319-4c01-83fe-0df57b41b69c\") " pod="openshift-ingress/e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm"
Nov 01 05:33:21.556513 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:33:21.556430    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:33:21.650047 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:33:21.650023    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm"
Nov 01 05:33:22.084368 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:33:22.084346    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm"]
Nov 01 05:33:22.086485 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:33:22.086443    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7183c3f_d319_4c01_83fe_0df57b41b69c.slice/crio-f0e66f4e70220eb59dc5240d97c7314fb57fdd285328541e825d92702a0b4527 WatchSource:0}: Error finding container f0e66f4e70220eb59dc5240d97c7314fb57fdd285328541e825d92702a0b4527: Status 404 returned error can't find the container with id f0e66f4e70220eb59dc5240d97c7314fb57fdd285328541e825d92702a0b4527
Nov 01 05:33:22.088748 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:33:22.088717    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:33:22.088824 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:33:22.088779    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:33:22.088824 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:33:22.088807    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:33:22.113134 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:33:22.113112    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm" event={"ID":"b7183c3f-d319-4c01-83fe-0df57b41b69c","Type":"ContainerStarted","Data":"f0e66f4e70220eb59dc5240d97c7314fb57fdd285328541e825d92702a0b4527"}
Nov 01 05:33:23.120155 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:33:23.120119    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm" event={"ID":"b7183c3f-d319-4c01-83fe-0df57b41b69c","Type":"ContainerStarted","Data":"f6f04e2a6248b9e4b0e00812f8807e1ec4d084072d7bdb2481a48e70de33447a"}
Nov 01 05:33:23.341812 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:33:23.341748    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm" podStartSLOduration=3.341728996 podStartE2EDuration="3.341728996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:33:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:33:23.289873884 +0000 UTC m=+1781.360670605" watchObservedRunningTime="2026-11-01 05:33:23.341728996 +0000 UTC m=+1781.412525718"
Nov 01 05:33:23.650918 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:33:23.650887    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm"
Nov 01 05:33:23.656235 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:33:23.656213    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm"
Nov 01 05:33:24.125107 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:33:24.125033    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm"
Nov 01 05:33:24.125848 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:33:24.125830    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm"
Nov 01 05:33:24.848552 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:33:24.848486    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:33:26.159526 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:33:26.159501    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:33:26.344423 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:33:26.344389    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:33:26.523365 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:33:26.523277    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:33:29.944743 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:33:29.944713    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:33:30.116046 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:33:30.115986    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:33:30.124551 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:33:30.124519    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:33:30.311177 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:33:30.311068    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:33:30.502627 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:33:30.502591    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:33:35.120829 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:33:35.120794    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:33:39.946179 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:33:39.946141    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:33:44.399205 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:33:44.399173    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:33:44.891003 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:33:44.890922    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:33:45.195372 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:33:45.195337    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:33:49.764017 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:33:49.763978    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:33:51.384957 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:33:51.384923    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:33:54.499815 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:33:54.499776    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:33:59.300914 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:33:59.300884    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:34:00.069439 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:34:00.069408    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:34:03.946966 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:03.946934    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:34:04.125160 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:04.125124    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:34:07.532283 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:07.532252    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:34:08.931135 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:08.931100    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:34:13.441799 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:13.441768    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1c4191-76f6bc474d-m8ljd"]
Nov 01 05:34:13.445775 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:13.445748    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1c4191-76f6bc474d-m8ljd"
Nov 01 05:34:13.497912 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:13.497879    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-1c4191-serving-cert\""
Nov 01 05:34:13.545051 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:13.545006    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9880a0a-c2b1-4a5e-a7b7-967038e13379-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-1c4191-76f6bc474d-m8ljd\" (UID: \"b9880a0a-c2b1-4a5e-a7b7-967038e13379\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1c4191-76f6bc474d-m8ljd"
Nov 01 05:34:13.545216 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:13.545056    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9880a0a-c2b1-4a5e-a7b7-967038e13379-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-1c4191-76f6bc474d-m8ljd\" (UID: \"b9880a0a-c2b1-4a5e-a7b7-967038e13379\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1c4191-76f6bc474d-m8ljd"
Nov 01 05:34:13.545344 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:13.545327    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9880a0a-c2b1-4a5e-a7b7-967038e13379-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-1c4191-76f6bc474d-m8ljd\" (UID: \"b9880a0a-c2b1-4a5e-a7b7-967038e13379\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1c4191-76f6bc474d-m8ljd"
Nov 01 05:34:13.545392 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:13.545371    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96pxn\" (UniqueName: \"kubernetes.io/projected/b9880a0a-c2b1-4a5e-a7b7-967038e13379-kube-api-access-96pxn\") pod \"maas-api-e2e-shared-b-w0-1c4191-76f6bc474d-m8ljd\" (UID: \"b9880a0a-c2b1-4a5e-a7b7-967038e13379\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1c4191-76f6bc474d-m8ljd"
Nov 01 05:34:13.550334 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:13.550312    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1c4191-76f6bc474d-m8ljd"]
Nov 01 05:34:13.646544 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:13.646514    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9880a0a-c2b1-4a5e-a7b7-967038e13379-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-1c4191-76f6bc474d-m8ljd\" (UID: \"b9880a0a-c2b1-4a5e-a7b7-967038e13379\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1c4191-76f6bc474d-m8ljd"
Nov 01 05:34:13.646718 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:13.646561    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96pxn\" (UniqueName: \"kubernetes.io/projected/b9880a0a-c2b1-4a5e-a7b7-967038e13379-kube-api-access-96pxn\") pod \"maas-api-e2e-shared-b-w0-1c4191-76f6bc474d-m8ljd\" (UID: \"b9880a0a-c2b1-4a5e-a7b7-967038e13379\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1c4191-76f6bc474d-m8ljd"
Nov 01 05:34:13.646718 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:13.646609    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9880a0a-c2b1-4a5e-a7b7-967038e13379-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-1c4191-76f6bc474d-m8ljd\" (UID: \"b9880a0a-c2b1-4a5e-a7b7-967038e13379\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1c4191-76f6bc474d-m8ljd"
Nov 01 05:34:13.646718 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:13.646629    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9880a0a-c2b1-4a5e-a7b7-967038e13379-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-1c4191-76f6bc474d-m8ljd\" (UID: \"b9880a0a-c2b1-4a5e-a7b7-967038e13379\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1c4191-76f6bc474d-m8ljd"
Nov 01 05:34:13.646888 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:34:13.646743    5689 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1c4191-serving-cert: secret "maas-api-e2e-shared-b-w0-1c4191-serving-cert" not found
Nov 01 05:34:13.646888 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:34:13.646820    5689 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9880a0a-c2b1-4a5e-a7b7-967038e13379-maas-api-tls podName:b9880a0a-c2b1-4a5e-a7b7-967038e13379 nodeName:}" failed. No retries permitted until 2026-11-01 05:34:14.146798042 +0000 UTC m=+1832.217594741 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b9880a0a-c2b1-4a5e-a7b7-967038e13379-maas-api-tls") pod "maas-api-e2e-shared-b-w0-1c4191-76f6bc474d-m8ljd" (UID: "b9880a0a-c2b1-4a5e-a7b7-967038e13379") : secret "maas-api-e2e-shared-b-w0-1c4191-serving-cert" not found
Nov 01 05:34:13.647223 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:13.647204    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9880a0a-c2b1-4a5e-a7b7-967038e13379-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-1c4191-76f6bc474d-m8ljd\" (UID: \"b9880a0a-c2b1-4a5e-a7b7-967038e13379\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1c4191-76f6bc474d-m8ljd"
Nov 01 05:34:13.649107 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:13.649089    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9880a0a-c2b1-4a5e-a7b7-967038e13379-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-1c4191-76f6bc474d-m8ljd\" (UID: \"b9880a0a-c2b1-4a5e-a7b7-967038e13379\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1c4191-76f6bc474d-m8ljd"
Nov 01 05:34:13.665874 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:13.665846    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:34:13.718426 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:13.718312    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96pxn\" (UniqueName: \"kubernetes.io/projected/b9880a0a-c2b1-4a5e-a7b7-967038e13379-kube-api-access-96pxn\") pod \"maas-api-e2e-shared-b-w0-1c4191-76f6bc474d-m8ljd\" (UID: \"b9880a0a-c2b1-4a5e-a7b7-967038e13379\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1c4191-76f6bc474d-m8ljd"
Nov 01 05:34:13.848380 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:13.848303    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:34:14.030312 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:14.030236    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:34:14.153992 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:14.153948    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9880a0a-c2b1-4a5e-a7b7-967038e13379-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-1c4191-76f6bc474d-m8ljd\" (UID: \"b9880a0a-c2b1-4a5e-a7b7-967038e13379\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1c4191-76f6bc474d-m8ljd"
Nov 01 05:34:14.156441 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:14.156415    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9880a0a-c2b1-4a5e-a7b7-967038e13379-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-1c4191-76f6bc474d-m8ljd\" (UID: \"b9880a0a-c2b1-4a5e-a7b7-967038e13379\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1c4191-76f6bc474d-m8ljd"
Nov 01 05:34:14.359654 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:14.359577    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1c4191-76f6bc474d-m8ljd"
Nov 01 05:34:14.738405 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:14.738378    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1c4191-76f6bc474d-m8ljd"]
Nov 01 05:34:14.740816 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:34:14.740792    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9880a0a_c2b1_4a5e_a7b7_967038e13379.slice/crio-936a6a78876940c8dc01cd8c12661ed4720493de4ceb2fc3f15677fa79695c1d WatchSource:0}: Error finding container 936a6a78876940c8dc01cd8c12661ed4720493de4ceb2fc3f15677fa79695c1d: Status 404 returned error can't find the container with id 936a6a78876940c8dc01cd8c12661ed4720493de4ceb2fc3f15677fa79695c1d
Nov 01 05:34:14.742628 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:14.742610    5689 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 05:34:14.915436 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:14.915394    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-kf57z"]
Nov 01 05:34:14.915891 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:14.915655    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-kf57z" podUID="eb42038d-b688-41da-9700-955bbda47478" containerName="istio-proxy" containerID="cri-o://ed565d2179bb0eee7829320de6b23f4a7c3732ad2ab158f4075a8a99d932a6c5" gracePeriod=30
Nov 01 05:34:14.964380 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:14.964348    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-p6p56"]
Nov 01 05:34:14.964602 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:14.964577    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-p6p56" podUID="fc051a97-b093-4019-957f-63b017c153a5" containerName="istio-proxy" containerID="cri-o://5803d671cf007499e141b4e69ac378d09c84b60af4294f801da5184a5e76e068" gracePeriod=30
Nov 01 05:34:15.420107 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:15.420075    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1c4191-76f6bc474d-m8ljd" event={"ID":"b9880a0a-c2b1-4a5e-a7b7-967038e13379","Type":"ContainerStarted","Data":"936a6a78876940c8dc01cd8c12661ed4720493de4ceb2fc3f15677fa79695c1d"}
Nov 01 05:34:15.916085 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:15.916055    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr"]
Nov 01 05:34:15.916569 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:15.916259    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr" podUID="91018a9a-6974-4492-8209-28c8c76ec6f7" containerName="istio-proxy" containerID="cri-o://3fda437e1eedd5bee56af00e45a214d3e2c3a53a6c25b8d3618baf2126003694" gracePeriod=30
Nov 01 05:34:16.427390 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:16.427345    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1c4191-76f6bc474d-m8ljd" event={"ID":"b9880a0a-c2b1-4a5e-a7b7-967038e13379","Type":"ContainerStarted","Data":"c88fc06982775064416be7d5a63ea00dfa4f6c024afec313f6df88024daeec37"}
Nov 01 05:34:16.427588 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:16.427479    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1c4191-76f6bc474d-m8ljd"
Nov 01 05:34:16.548118 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:16.548045    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1c4191-76f6bc474d-m8ljd" podStartSLOduration=2.76481321 podStartE2EDuration="3.548029051s" totalImagesPullingTime="783.215841ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:34:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:34:16.545965655 +0000 UTC m=+1834.616762375" watchObservedRunningTime="2026-11-01 05:34:16.548029051 +0000 UTC m=+1834.618825771"
Nov 01 05:34:16.695474 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:16.695133    5689 patch_prober.go:29] interesting pod/e2e-worker-w4-a48099-openshift-default-577864759-p6p56 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 05:34:16.695646 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:16.695581    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-p6p56" podUID="fc051a97-b093-4019-957f-63b017c153a5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 05:34:18.212952 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:18.212918    5689 patch_prober.go:29] interesting pod/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 05:34:18.213311 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:18.212975    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr" podUID="91018a9a-6974-4492-8209-28c8c76ec6f7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 05:34:18.716712 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:18.716678    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:34:18.983495 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:18.983399    5689 patch_prober.go:29] interesting pod/e2e-worker-w4-a48099-openshift-default-577864759-kf57z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 05:34:18.983495 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:18.983469    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-kf57z" podUID="eb42038d-b688-41da-9700-955bbda47478" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 05:34:20.243283 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:20.243253    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-kf57z"
Nov 01 05:34:20.286725 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:20.286700    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-p6p56"
Nov 01 05:34:20.361373 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:20.361345    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb42038d-b688-41da-9700-955bbda47478-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb42038d-b688-41da-9700-955bbda47478-istio-data\") pod \"eb42038d-b688-41da-9700-955bbda47478\" (UID: \"eb42038d-b688-41da-9700-955bbda47478\") "
Nov 01 05:34:20.361593 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:20.361390    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb42038d-b688-41da-9700-955bbda47478-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb42038d-b688-41da-9700-955bbda47478-istiod-ca-cert\") pod \"eb42038d-b688-41da-9700-955bbda47478\" (UID: \"eb42038d-b688-41da-9700-955bbda47478\") "
Nov 01 05:34:20.361593 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:20.361413    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb42038d-b688-41da-9700-955bbda47478-istio-token\" (UniqueName: \"kubernetes.io/projected/eb42038d-b688-41da-9700-955bbda47478-istio-token\") pod \"eb42038d-b688-41da-9700-955bbda47478\" (UID: \"eb42038d-b688-41da-9700-955bbda47478\") "
Nov 01 05:34:20.361593 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:20.361501    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb42038d-b688-41da-9700-955bbda47478-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb42038d-b688-41da-9700-955bbda47478-credential-socket\") pod \"eb42038d-b688-41da-9700-955bbda47478\" (UID: \"eb42038d-b688-41da-9700-955bbda47478\") "
Nov 01 05:34:20.361772 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:20.361637    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb42038d-b688-41da-9700-955bbda47478-kube-api-access-v9rhw\" (UniqueName: \"kubernetes.io/projected/eb42038d-b688-41da-9700-955bbda47478-kube-api-access-v9rhw\") pod \"eb42038d-b688-41da-9700-955bbda47478\" (UID: \"eb42038d-b688-41da-9700-955bbda47478\") "
Nov 01 05:34:20.361772 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:20.361672    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb42038d-b688-41da-9700-955bbda47478-istio-data" pod "eb42038d-b688-41da-9700-955bbda47478" (UID: "eb42038d-b688-41da-9700-955bbda47478"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:34:20.361772 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:20.361691    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb42038d-b688-41da-9700-955bbda47478-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb42038d-b688-41da-9700-955bbda47478-workload-certs\") pod \"eb42038d-b688-41da-9700-955bbda47478\" (UID: \"eb42038d-b688-41da-9700-955bbda47478\") "
Nov 01 05:34:20.361940 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:20.361774    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb42038d-b688-41da-9700-955bbda47478-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb42038d-b688-41da-9700-955bbda47478-istio-envoy\") pod \"eb42038d-b688-41da-9700-955bbda47478\" (UID: \"eb42038d-b688-41da-9700-955bbda47478\") "
Nov 01 05:34:20.361940 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:20.361808    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eb42038d-b688-41da-9700-955bbda47478-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb42038d-b688-41da-9700-955bbda47478-istio-podinfo\") pod \"eb42038d-b688-41da-9700-955bbda47478\" (UID: \"eb42038d-b688-41da-9700-955bbda47478\") "
Nov 01 05:34:20.361940 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:20.361838    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb42038d-b688-41da-9700-955bbda47478-credential-socket" pod "eb42038d-b688-41da-9700-955bbda47478" (UID: "eb42038d-b688-41da-9700-955bbda47478"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:34:20.361940 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:20.361861    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb42038d-b688-41da-9700-955bbda47478-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb42038d-b688-41da-9700-955bbda47478-workload-socket\") pod \"eb42038d-b688-41da-9700-955bbda47478\" (UID: \"eb42038d-b688-41da-9700-955bbda47478\") "
Nov 01 05:34:20.361940 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:20.361867    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb42038d-b688-41da-9700-955bbda47478-istiod-ca-cert" pod "eb42038d-b688-41da-9700-955bbda47478" (UID: "eb42038d-b688-41da-9700-955bbda47478"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:34:20.362196 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:20.361987    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb42038d-b688-41da-9700-955bbda47478-workload-certs" pod "eb42038d-b688-41da-9700-955bbda47478" (UID: "eb42038d-b688-41da-9700-955bbda47478"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:34:20.362196 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:20.362142    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb42038d-b688-41da-9700-955bbda47478-workload-socket" pod "eb42038d-b688-41da-9700-955bbda47478" (UID: "eb42038d-b688-41da-9700-955bbda47478"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:34:20.362626 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:20.362510    5689 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb42038d-b688-41da-9700-955bbda47478-credential-socket\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:34:20.362762 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:20.362640    5689 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb42038d-b688-41da-9700-955bbda47478-workload-certs\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:34:20.362762 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:20.362655    5689 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb42038d-b688-41da-9700-955bbda47478-workload-socket\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:34:20.362762 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:20.362669    5689 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb42038d-b688-41da-9700-955bbda47478-istio-data\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:34:20.362762 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:20.362684    5689 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb42038d-b688-41da-9700-955bbda47478-istiod-ca-cert\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:34:20.363836 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:20.363811    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb42038d-b688-41da-9700-955bbda47478-istio-token" pod "eb42038d-b688-41da-9700-955bbda47478" (UID: "eb42038d-b688-41da-9700-955bbda47478"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:34:20.363941 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:20.363879    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb42038d-b688-41da-9700-955bbda47478-kube-api-access-v9rhw" pod "eb42038d-b688-41da-9700-955bbda47478" (UID: "eb42038d-b688-41da-9700-955bbda47478"). InnerVolumeSpecName "kube-api-access-v9rhw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:34:20.364002 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:20.363944    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eb42038d-b688-41da-9700-955bbda47478-istio-podinfo" pod "eb42038d-b688-41da-9700-955bbda47478" (UID: "eb42038d-b688-41da-9700-955bbda47478"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 05:34:20.364061 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:20.364044    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb42038d-b688-41da-9700-955bbda47478-istio-envoy" pod "eb42038d-b688-41da-9700-955bbda47478" (UID: "eb42038d-b688-41da-9700-955bbda47478"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:34:20.457827 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:20.457733    5689 generic.go:356] "Generic (PLEG): container finished" podID="eb42038d-b688-41da-9700-955bbda47478" containerID="ed565d2179bb0eee7829320de6b23f4a7c3732ad2ab158f4075a8a99d932a6c5" exitCode=0
Nov 01 05:34:20.457827 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:20.457811    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-kf57z"
Nov 01 05:34:20.457987 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:20.457909    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-kf57z" event={"ID":"eb42038d-b688-41da-9700-955bbda47478","Type":"ContainerDied","Data":"ed565d2179bb0eee7829320de6b23f4a7c3732ad2ab158f4075a8a99d932a6c5"}
Nov 01 05:34:20.457987 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:20.457965    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-kf57z" event={"ID":"eb42038d-b688-41da-9700-955bbda47478","Type":"ContainerDied","Data":"1360db150d11086e20a0c3adb6a63ea6c7c0b4f8b2e7ff46eb8b63031f131d49"}
Nov 01 05:34:20.458072 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:20.457993    5689 scope.go:122] "RemoveContainer" containerID="ed565d2179bb0eee7829320de6b23f4a7c3732ad2ab158f4075a8a99d932a6c5"
Nov 01 05:34:20.459968 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:20.459938    5689 generic.go:356] "Generic (PLEG): container finished" podID="fc051a97-b093-4019-957f-63b017c153a5" containerID="5803d671cf007499e141b4e69ac378d09c84b60af4294f801da5184a5e76e068" exitCode=0
Nov 01 05:34:20.460101 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:20.459985    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-p6p56"
Nov 01 05:34:20.460101 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:20.460012    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-p6p56" event={"ID":"fc051a97-b093-4019-957f-63b017c153a5","Type":"ContainerDied","Data":"5803d671cf007499e141b4e69ac378d09c84b60af4294f801da5184a5e76e068"}
Nov 01 05:34:20.460101 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:20.460061    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-p6p56" event={"ID":"fc051a97-b093-4019-957f-63b017c153a5","Type":"ContainerDied","Data":"05ef50fb96737c173b4019a520c10afb4882dd61f54cf25a25a633ab316de8ce"}
Nov 01 05:34:20.463857 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:20.463191    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc051a97-b093-4019-957f-63b017c153a5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc051a97-b093-4019-957f-63b017c153a5-istio-data\") pod \"fc051a97-b093-4019-957f-63b017c153a5\" (UID: \"fc051a97-b093-4019-957f-63b017c153a5\") "
Nov 01 05:34:20.463857 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:20.463245    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc051a97-b093-4019-957f-63b017c153a5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc051a97-b093-4019-957f-63b017c153a5-workload-socket\") pod \"fc051a97-b093-4019-957f-63b017c153a5\" (UID: \"fc051a97-b093-4019-957f-63b017c153a5\") "
Nov 01 05:34:20.463857 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:20.463301    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc051a97-b093-4019-957f-63b017c153a5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc051a97-b093-4019-957f-63b017c153a5-istio-envoy\") pod \"fc051a97-b093-4019-957f-63b017c153a5\" (UID: \"fc051a97-b093-4019-957f-63b017c153a5\") "
Nov 01 05:34:20.463857 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:20.463417    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc051a97-b093-4019-957f-63b017c153a5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc051a97-b093-4019-957f-63b017c153a5-workload-certs\") pod \"fc051a97-b093-4019-957f-63b017c153a5\" (UID: \"fc051a97-b093-4019-957f-63b017c153a5\") "
Nov 01 05:34:20.463857 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:20.463445    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc051a97-b093-4019-957f-63b017c153a5-kube-api-access-zjmcs\" (UniqueName: \"kubernetes.io/projected/fc051a97-b093-4019-957f-63b017c153a5-kube-api-access-zjmcs\") pod \"fc051a97-b093-4019-957f-63b017c153a5\" (UID: \"fc051a97-b093-4019-957f-63b017c153a5\") "
Nov 01 05:34:20.463857 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:20.463544    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc051a97-b093-4019-957f-63b017c153a5-istio-data" pod "fc051a97-b093-4019-957f-63b017c153a5" (UID: "fc051a97-b093-4019-957f-63b017c153a5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:34:20.463857 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:20.463550    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fc051a97-b093-4019-957f-63b017c153a5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc051a97-b093-4019-957f-63b017c153a5-istio-podinfo\") pod \"fc051a97-b093-4019-957f-63b017c153a5\" (UID: \"fc051a97-b093-4019-957f-63b017c153a5\") "
Nov 01 05:34:20.463857 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:20.463646    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc051a97-b093-4019-957f-63b017c153a5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc051a97-b093-4019-957f-63b017c153a5-credential-socket\") pod \"fc051a97-b093-4019-957f-63b017c153a5\" (UID: \"fc051a97-b093-4019-957f-63b017c153a5\") "
Nov 01 05:34:20.463857 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:20.463700    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc051a97-b093-4019-957f-63b017c153a5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc051a97-b093-4019-957f-63b017c153a5-istiod-ca-cert\") pod \"fc051a97-b093-4019-957f-63b017c153a5\" (UID: \"fc051a97-b093-4019-957f-63b017c153a5\") "
Nov 01 05:34:20.463857 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:20.463731    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc051a97-b093-4019-957f-63b017c153a5-istio-token\" (UniqueName: \"kubernetes.io/projected/fc051a97-b093-4019-957f-63b017c153a5-istio-token\") pod \"fc051a97-b093-4019-957f-63b017c153a5\" (UID: \"fc051a97-b093-4019-957f-63b017c153a5\") "
Nov 01 05:34:20.464400 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:20.464116    5689 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb42038d-b688-41da-9700-955bbda47478-istio-token\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:34:20.464400 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:20.464128    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9rhw\" (UniqueName: \"kubernetes.io/projected/eb42038d-b688-41da-9700-955bbda47478-kube-api-access-v9rhw\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:34:20.464400 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:20.464140    5689 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb42038d-b688-41da-9700-955bbda47478-istio-envoy\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:34:20.464400 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:20.464148    5689 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb42038d-b688-41da-9700-955bbda47478-istio-podinfo\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:34:20.464400 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:20.464156    5689 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc051a97-b093-4019-957f-63b017c153a5-istio-data\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:34:20.464921 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:20.464897    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc051a97-b093-4019-957f-63b017c153a5-workload-certs" pod "fc051a97-b093-4019-957f-63b017c153a5" (UID: "fc051a97-b093-4019-957f-63b017c153a5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:34:20.465829 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:20.465294    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc051a97-b093-4019-957f-63b017c153a5-workload-socket" pod "fc051a97-b093-4019-957f-63b017c153a5" (UID: "fc051a97-b093-4019-957f-63b017c153a5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:34:20.465829 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:20.465494    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc051a97-b093-4019-957f-63b017c153a5-credential-socket" pod "fc051a97-b093-4019-957f-63b017c153a5" (UID: "fc051a97-b093-4019-957f-63b017c153a5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:34:20.466022 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:20.465831    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc051a97-b093-4019-957f-63b017c153a5-istiod-ca-cert" pod "fc051a97-b093-4019-957f-63b017c153a5" (UID: "fc051a97-b093-4019-957f-63b017c153a5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:34:20.468652 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:20.468561    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc051a97-b093-4019-957f-63b017c153a5-istio-envoy" pod "fc051a97-b093-4019-957f-63b017c153a5" (UID: "fc051a97-b093-4019-957f-63b017c153a5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:34:20.468652 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:20.468593    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc051a97-b093-4019-957f-63b017c153a5-istio-token" pod "fc051a97-b093-4019-957f-63b017c153a5" (UID: "fc051a97-b093-4019-957f-63b017c153a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:34:20.468652 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:20.468612    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc051a97-b093-4019-957f-63b017c153a5-kube-api-access-zjmcs" pod "fc051a97-b093-4019-957f-63b017c153a5" (UID: "fc051a97-b093-4019-957f-63b017c153a5"). InnerVolumeSpecName "kube-api-access-zjmcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:34:20.468652 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:20.468623    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fc051a97-b093-4019-957f-63b017c153a5-istio-podinfo" pod "fc051a97-b093-4019-957f-63b017c153a5" (UID: "fc051a97-b093-4019-957f-63b017c153a5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 05:34:20.470138 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:20.470108    5689 scope.go:122] "RemoveContainer" containerID="ed565d2179bb0eee7829320de6b23f4a7c3732ad2ab158f4075a8a99d932a6c5"
Nov 01 05:34:20.470397 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:34:20.470378    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed565d2179bb0eee7829320de6b23f4a7c3732ad2ab158f4075a8a99d932a6c5\": container with ID starting with ed565d2179bb0eee7829320de6b23f4a7c3732ad2ab158f4075a8a99d932a6c5 not found: ID does not exist" containerID="ed565d2179bb0eee7829320de6b23f4a7c3732ad2ab158f4075a8a99d932a6c5"
Nov 01 05:34:20.470464 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:20.470403    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed565d2179bb0eee7829320de6b23f4a7c3732ad2ab158f4075a8a99d932a6c5"} err="failed to get container status \"ed565d2179bb0eee7829320de6b23f4a7c3732ad2ab158f4075a8a99d932a6c5\": rpc error: code = NotFound desc = could not find container \"ed565d2179bb0eee7829320de6b23f4a7c3732ad2ab158f4075a8a99d932a6c5\": container with ID starting with ed565d2179bb0eee7829320de6b23f4a7c3732ad2ab158f4075a8a99d932a6c5 not found: ID does not exist"
Nov 01 05:34:20.470464 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:20.470419    5689 scope.go:122] "RemoveContainer" containerID="5803d671cf007499e141b4e69ac378d09c84b60af4294f801da5184a5e76e068"
Nov 01 05:34:20.479316 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:20.479299    5689 scope.go:122] "RemoveContainer" containerID="5803d671cf007499e141b4e69ac378d09c84b60af4294f801da5184a5e76e068"
Nov 01 05:34:20.479586 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:34:20.479567    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5803d671cf007499e141b4e69ac378d09c84b60af4294f801da5184a5e76e068\": container with ID starting with 5803d671cf007499e141b4e69ac378d09c84b60af4294f801da5184a5e76e068 not found: ID does not exist" containerID="5803d671cf007499e141b4e69ac378d09c84b60af4294f801da5184a5e76e068"
Nov 01 05:34:20.479670 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:20.479589    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5803d671cf007499e141b4e69ac378d09c84b60af4294f801da5184a5e76e068"} err="failed to get container status \"5803d671cf007499e141b4e69ac378d09c84b60af4294f801da5184a5e76e068\": rpc error: code = NotFound desc = could not find container \"5803d671cf007499e141b4e69ac378d09c84b60af4294f801da5184a5e76e068\": container with ID starting with 5803d671cf007499e141b4e69ac378d09c84b60af4294f801da5184a5e76e068 not found: ID does not exist"
Nov 01 05:34:20.565857 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:20.565829    5689 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc051a97-b093-4019-957f-63b017c153a5-workload-certs\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:34:20.565857 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:20.565854    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjmcs\" (UniqueName: \"kubernetes.io/projected/fc051a97-b093-4019-957f-63b017c153a5-kube-api-access-zjmcs\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:34:20.565857 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:20.565864    5689 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc051a97-b093-4019-957f-63b017c153a5-istio-podinfo\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:34:20.566113 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:20.565874    5689 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc051a97-b093-4019-957f-63b017c153a5-credential-socket\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:34:20.566113 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:20.565885    5689 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc051a97-b093-4019-957f-63b017c153a5-istiod-ca-cert\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:34:20.566113 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:20.565893    5689 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc051a97-b093-4019-957f-63b017c153a5-istio-token\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:34:20.566113 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:20.565902    5689 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc051a97-b093-4019-957f-63b017c153a5-workload-socket\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:34:20.566113 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:20.565910    5689 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc051a97-b093-4019-957f-63b017c153a5-istio-envoy\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:34:20.699376 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:20.699347    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-kf57z"]
Nov 01 05:34:20.704057 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:20.704018    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-kf57z"]
Nov 01 05:34:20.883277 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:20.883241    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-p6p56"]
Nov 01 05:34:20.887368 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:20.887345    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-p6p56"]
Nov 01 05:34:21.183081 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:21.183060    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr"
Nov 01 05:34:21.273610 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:21.273582    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91018a9a-6974-4492-8209-28c8c76ec6f7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91018a9a-6974-4492-8209-28c8c76ec6f7-istiod-ca-cert\") pod \"91018a9a-6974-4492-8209-28c8c76ec6f7\" (UID: \"91018a9a-6974-4492-8209-28c8c76ec6f7\") "
Nov 01 05:34:21.273979 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:21.273636    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91018a9a-6974-4492-8209-28c8c76ec6f7-kube-api-access-r85zj\" (UniqueName: \"kubernetes.io/projected/91018a9a-6974-4492-8209-28c8c76ec6f7-kube-api-access-r85zj\") pod \"91018a9a-6974-4492-8209-28c8c76ec6f7\" (UID: \"91018a9a-6974-4492-8209-28c8c76ec6f7\") "
Nov 01 05:34:21.273979 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:21.273668    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91018a9a-6974-4492-8209-28c8c76ec6f7-istio-token\" (UniqueName: \"kubernetes.io/projected/91018a9a-6974-4492-8209-28c8c76ec6f7-istio-token\") pod \"91018a9a-6974-4492-8209-28c8c76ec6f7\" (UID: \"91018a9a-6974-4492-8209-28c8c76ec6f7\") "
Nov 01 05:34:21.273979 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:21.273709    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91018a9a-6974-4492-8209-28c8c76ec6f7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91018a9a-6974-4492-8209-28c8c76ec6f7-workload-socket\") pod \"91018a9a-6974-4492-8209-28c8c76ec6f7\" (UID: \"91018a9a-6974-4492-8209-28c8c76ec6f7\") "
Nov 01 05:34:21.273979 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:21.273796    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91018a9a-6974-4492-8209-28c8c76ec6f7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91018a9a-6974-4492-8209-28c8c76ec6f7-credential-socket\") pod \"91018a9a-6974-4492-8209-28c8c76ec6f7\" (UID: \"91018a9a-6974-4492-8209-28c8c76ec6f7\") "
Nov 01 05:34:21.273979 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:21.273833    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/91018a9a-6974-4492-8209-28c8c76ec6f7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91018a9a-6974-4492-8209-28c8c76ec6f7-istio-podinfo\") pod \"91018a9a-6974-4492-8209-28c8c76ec6f7\" (UID: \"91018a9a-6974-4492-8209-28c8c76ec6f7\") "
Nov 01 05:34:21.273979 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:21.273870    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91018a9a-6974-4492-8209-28c8c76ec6f7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91018a9a-6974-4492-8209-28c8c76ec6f7-istio-envoy\") pod \"91018a9a-6974-4492-8209-28c8c76ec6f7\" (UID: \"91018a9a-6974-4492-8209-28c8c76ec6f7\") "
Nov 01 05:34:21.273979 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:21.273906    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91018a9a-6974-4492-8209-28c8c76ec6f7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91018a9a-6974-4492-8209-28c8c76ec6f7-workload-certs\") pod \"91018a9a-6974-4492-8209-28c8c76ec6f7\" (UID: \"91018a9a-6974-4492-8209-28c8c76ec6f7\") "
Nov 01 05:34:21.273979 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:21.273966    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91018a9a-6974-4492-8209-28c8c76ec6f7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91018a9a-6974-4492-8209-28c8c76ec6f7-istio-data\") pod \"91018a9a-6974-4492-8209-28c8c76ec6f7\" (UID: \"91018a9a-6974-4492-8209-28c8c76ec6f7\") "
Nov 01 05:34:21.274661 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:21.274004    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91018a9a-6974-4492-8209-28c8c76ec6f7-istiod-ca-cert" pod "91018a9a-6974-4492-8209-28c8c76ec6f7" (UID: "91018a9a-6974-4492-8209-28c8c76ec6f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:34:21.274661 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:21.274125    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91018a9a-6974-4492-8209-28c8c76ec6f7-workload-socket" pod "91018a9a-6974-4492-8209-28c8c76ec6f7" (UID: "91018a9a-6974-4492-8209-28c8c76ec6f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:34:21.274661 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:21.274145    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91018a9a-6974-4492-8209-28c8c76ec6f7-credential-socket" pod "91018a9a-6974-4492-8209-28c8c76ec6f7" (UID: "91018a9a-6974-4492-8209-28c8c76ec6f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:34:21.274661 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:21.274307    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91018a9a-6974-4492-8209-28c8c76ec6f7-istio-data" pod "91018a9a-6974-4492-8209-28c8c76ec6f7" (UID: "91018a9a-6974-4492-8209-28c8c76ec6f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:34:21.274661 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:21.274329    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91018a9a-6974-4492-8209-28c8c76ec6f7-workload-certs" pod "91018a9a-6974-4492-8209-28c8c76ec6f7" (UID: "91018a9a-6974-4492-8209-28c8c76ec6f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:34:21.274941 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:21.274675    5689 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91018a9a-6974-4492-8209-28c8c76ec6f7-workload-socket\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:34:21.274941 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:21.274696    5689 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91018a9a-6974-4492-8209-28c8c76ec6f7-credential-socket\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:34:21.274941 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:21.274711    5689 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91018a9a-6974-4492-8209-28c8c76ec6f7-workload-certs\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:34:21.274941 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:21.274725    5689 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91018a9a-6974-4492-8209-28c8c76ec6f7-istio-data\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:34:21.274941 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:21.274740    5689 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91018a9a-6974-4492-8209-28c8c76ec6f7-istiod-ca-cert\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:34:21.276157 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:21.276129    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91018a9a-6974-4492-8209-28c8c76ec6f7-kube-api-access-r85zj" pod "91018a9a-6974-4492-8209-28c8c76ec6f7" (UID: "91018a9a-6974-4492-8209-28c8c76ec6f7"). InnerVolumeSpecName "kube-api-access-r85zj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:34:21.276157 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:21.276147    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91018a9a-6974-4492-8209-28c8c76ec6f7-istio-token" pod "91018a9a-6974-4492-8209-28c8c76ec6f7" (UID: "91018a9a-6974-4492-8209-28c8c76ec6f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:34:21.276317 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:21.276188    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91018a9a-6974-4492-8209-28c8c76ec6f7-istio-envoy" pod "91018a9a-6974-4492-8209-28c8c76ec6f7" (UID: "91018a9a-6974-4492-8209-28c8c76ec6f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:34:21.276354 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:21.276313    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/91018a9a-6974-4492-8209-28c8c76ec6f7-istio-podinfo" pod "91018a9a-6974-4492-8209-28c8c76ec6f7" (UID: "91018a9a-6974-4492-8209-28c8c76ec6f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 05:34:21.376069 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:21.375994    5689 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91018a9a-6974-4492-8209-28c8c76ec6f7-istio-podinfo\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:34:21.376069 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:21.376021    5689 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91018a9a-6974-4492-8209-28c8c76ec6f7-istio-envoy\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:34:21.376069 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:21.376031    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r85zj\" (UniqueName: \"kubernetes.io/projected/91018a9a-6974-4492-8209-28c8c76ec6f7-kube-api-access-r85zj\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:34:21.376069 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:21.376040    5689 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91018a9a-6974-4492-8209-28c8c76ec6f7-istio-token\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:34:21.470653 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:21.470624    5689 generic.go:356] "Generic (PLEG): container finished" podID="91018a9a-6974-4492-8209-28c8c76ec6f7" containerID="3fda437e1eedd5bee56af00e45a214d3e2c3a53a6c25b8d3618baf2126003694" exitCode=0
Nov 01 05:34:21.470826 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:21.470676    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr"
Nov 01 05:34:21.470826 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:21.470685    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr" event={"ID":"91018a9a-6974-4492-8209-28c8c76ec6f7","Type":"ContainerDied","Data":"3fda437e1eedd5bee56af00e45a214d3e2c3a53a6c25b8d3618baf2126003694"}
Nov 01 05:34:21.470826 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:21.470726    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr" event={"ID":"91018a9a-6974-4492-8209-28c8c76ec6f7","Type":"ContainerDied","Data":"a88c78b8b3d0254f38ae4989112c3d5d09861ea3b688611d0d09ce5ff1b70610"}
Nov 01 05:34:21.470826 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:21.470747    5689 scope.go:122] "RemoveContainer" containerID="3fda437e1eedd5bee56af00e45a214d3e2c3a53a6c25b8d3618baf2126003694"
Nov 01 05:34:21.480595 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:21.480503    5689 scope.go:122] "RemoveContainer" containerID="3fda437e1eedd5bee56af00e45a214d3e2c3a53a6c25b8d3618baf2126003694"
Nov 01 05:34:21.480756 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:34:21.480738    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fda437e1eedd5bee56af00e45a214d3e2c3a53a6c25b8d3618baf2126003694\": container with ID starting with 3fda437e1eedd5bee56af00e45a214d3e2c3a53a6c25b8d3618baf2126003694 not found: ID does not exist" containerID="3fda437e1eedd5bee56af00e45a214d3e2c3a53a6c25b8d3618baf2126003694"
Nov 01 05:34:21.480804 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:21.480765    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fda437e1eedd5bee56af00e45a214d3e2c3a53a6c25b8d3618baf2126003694"} err="failed to get container status \"3fda437e1eedd5bee56af00e45a214d3e2c3a53a6c25b8d3618baf2126003694\": rpc error: code = NotFound desc = could not find container \"3fda437e1eedd5bee56af00e45a214d3e2c3a53a6c25b8d3618baf2126003694\": container with ID starting with 3fda437e1eedd5bee56af00e45a214d3e2c3a53a6c25b8d3618baf2126003694 not found: ID does not exist"
Nov 01 05:34:21.636387 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:21.636310    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr"]
Nov 01 05:34:21.640598 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:21.640565    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-4s6hr"]
Nov 01 05:34:22.068337 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:22.068295    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91018a9a-6974-4492-8209-28c8c76ec6f7" path="/var/lib/kubelet/pods/91018a9a-6974-4492-8209-28c8c76ec6f7/volumes"
Nov 01 05:34:22.068859 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:22.068836    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb42038d-b688-41da-9700-955bbda47478" path="/var/lib/kubelet/pods/eb42038d-b688-41da-9700-955bbda47478/volumes"
Nov 01 05:34:22.069323 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:22.069303    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc051a97-b093-4019-957f-63b017c153a5" path="/var/lib/kubelet/pods/fc051a97-b093-4019-957f-63b017c153a5/volumes"
Nov 01 05:34:22.439549 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:22.439521    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1c4191-76f6bc474d-m8ljd"
Nov 01 05:34:23.652929 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:23.652898    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:34:28.438535 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:28.438494    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:34:28.541858 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:28.541734    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:34:30.070786 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:34:30.070754    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:34:33.804261 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:33.804231    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:34:39.126522 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:39.126492    5689 patch_prober.go:29] interesting pod/e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Nov 01 05:34:39.126960 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:39.126553    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm" podUID="b7183c3f-d319-4c01-83fe-0df57b41b69c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Nov 01 05:34:39.146489 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:39.146438    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:34:39.576066 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:39.576030    5689 generic.go:356] "Generic (PLEG): container finished" podID="b7183c3f-d319-4c01-83fe-0df57b41b69c" containerID="f6f04e2a6248b9e4b0e00812f8807e1ec4d084072d7bdb2481a48e70de33447a" exitCode=0
Nov 01 05:34:39.576235 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:39.576109    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm" event={"ID":"b7183c3f-d319-4c01-83fe-0df57b41b69c","Type":"ContainerDied","Data":"f6f04e2a6248b9e4b0e00812f8807e1ec4d084072d7bdb2481a48e70de33447a"}
Nov 01 05:34:39.576518 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:39.576501    5689 scope.go:122] "RemoveContainer" containerID="f6f04e2a6248b9e4b0e00812f8807e1ec4d084072d7bdb2481a48e70de33447a"
Nov 01 05:34:39.578177 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:39.578150    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:34:39.578282 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:39.578216    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:34:39.578282 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:39.578246    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:34:40.603050 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:40.603016    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm" event={"ID":"b7183c3f-d319-4c01-83fe-0df57b41b69c","Type":"ContainerStarted","Data":"56283a243135e0b01e7c68283761ff9fa9d93a8b4537d04459a2df5c0605b3ec"}
Nov 01 05:34:40.651052 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:40.651022    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm"
Nov 01 05:34:40.651208 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:40.651151    5689 patch_prober.go:29] interesting pod/e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Nov 01 05:34:40.651208 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:40.651191    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm" podUID="b7183c3f-d319-4c01-83fe-0df57b41b69c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Nov 01 05:34:41.650345 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:41.650313    5689 patch_prober.go:29] interesting pod/e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Nov 01 05:34:41.650768 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:41.650372    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm" podUID="b7183c3f-d319-4c01-83fe-0df57b41b69c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Nov 01 05:34:42.651007 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:42.650971    5689 patch_prober.go:29] interesting pod/e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Nov 01 05:34:42.651398 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:42.651036    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm" podUID="b7183c3f-d319-4c01-83fe-0df57b41b69c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Nov 01 05:34:43.651313 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:43.651278    5689 patch_prober.go:29] interesting pod/e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Nov 01 05:34:43.651753 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:43.651346    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm" podUID="b7183c3f-d319-4c01-83fe-0df57b41b69c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Nov 01 05:34:43.900967 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:43.900924    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:34:44.651065 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:44.651026    5689 patch_prober.go:29] interesting pod/e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Nov 01 05:34:44.651273 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:44.651089    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm" podUID="b7183c3f-d319-4c01-83fe-0df57b41b69c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Nov 01 05:34:44.902491 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:44.902378    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:34:45.084106 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:45.084056    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:34:45.698392 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:45.698355    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm"
Nov 01 05:34:45.698905 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:45.698834    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm"
Nov 01 05:34:45.704294 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:45.704267    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm"
Nov 01 05:34:48.725503 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:48.715881    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:34:53.435153 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:53.435081    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:34:58.186633 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:34:58.186602    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:35:00.068884 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:35:00.068847    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:35:03.021284 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:35:03.021235    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:35:04.309432 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:35:04.309400    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:35:07.811573 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:35:07.811518    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:35:12.616644 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:35:12.616612    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:35:13.234079 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:35:13.234032    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:35:15.819947 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:35:15.819913    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:35:16.039996 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:35:16.039959    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:35:17.058759 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:35:17.058733    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:35:17.334995 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:35:17.334915    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:35:17.515072 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:35:17.515034    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:35:17.699475 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:35:17.696026    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:35:18.047000 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:35:18.046923    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90d244ee-g5tn4"]
Nov 01 05:35:18.096197 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:35:18.096168    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90d244ee-g5tn4"]
Nov 01 05:35:18.589001 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:35:18.588971    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:35:18.850611 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:35:18.850524    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:35:19.037685 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:35:19.037630    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:35:19.296578 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:35:19.296551    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:35:20.005412 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:35:20.005375    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:35:20.068168 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:35:20.068080    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8374e5fe-7d4d-4a3c-824a-6447cac92b99" path="/var/lib/kubelet/pods/8374e5fe-7d4d-4a3c-824a-6447cac92b99/volumes"
Nov 01 05:35:20.191389 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:35:20.191350    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:35:20.378684 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:35:20.378603    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:35:20.873304 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:35:20.873239    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:35:21.091902 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:35:21.091868    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:35:24.226011 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:35:24.225975    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:35:24.424199 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:35:24.424163    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:35:24.633668 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:35:24.633584    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:35:25.947852 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:35:25.947822    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:35:26.126343 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:35:26.126312    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:35:27.000245 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:35:27.000215    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:35:27.179062 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:35:27.179035    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:35:29.655838 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:35:29.655758    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:35:30.068003 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:35:30.067926    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:35:30.371359 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:35:30.371281    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:35:30.794752 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:35:30.794721    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:35:34.608967 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:35:34.608933    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:35:37.356391 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:35:37.356353    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:35:39.451930 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:35:39.451877    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:35:44.245128 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:35:44.245092    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:35:49.018194 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:35:49.018151    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:35:49.248872 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:35:49.248840    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:35:49.437962 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:35:49.437913    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:35:53.604750 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:35:53.604717    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-794cct6cj"]
Nov 01 05:35:53.605265 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:35:53.605251    5689 cpu_manager.go:414] "containerMap: removing container" podUID="eb42038d-b688-41da-9700-955bbda47478" containerName="istio-proxy"
Nov 01 05:35:53.605308 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:35:53.605267    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb42038d-b688-41da-9700-955bbda47478" containerName="istio-proxy"
Nov 01 05:35:53.605308 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:35:53.605291    5689 cpu_manager.go:414] "containerMap: removing container" podUID="fc051a97-b093-4019-957f-63b017c153a5" containerName="istio-proxy"
Nov 01 05:35:53.605308 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:35:53.605295    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc051a97-b093-4019-957f-63b017c153a5" containerName="istio-proxy"
Nov 01 05:35:53.605406 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:35:53.605314    5689 cpu_manager.go:414] "containerMap: removing container" podUID="91018a9a-6974-4492-8209-28c8c76ec6f7" containerName="istio-proxy"
Nov 01 05:35:53.605406 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:35:53.605321    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91018a9a-6974-4492-8209-28c8c76ec6f7" containerName="istio-proxy"
Nov 01 05:35:53.605494 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:35:53.605406    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb42038d-b688-41da-9700-955bbda47478" containerName="istio-proxy"
Nov 01 05:35:53.605494 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:35:53.605421    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc051a97-b093-4019-957f-63b017c153a5" containerName="istio-proxy"
Nov 01 05:35:53.605494 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:35:53.605430    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="91018a9a-6974-4492-8209-28c8c76ec6f7" containerName="istio-proxy"
Nov 01 05:35:53.610278 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:35:53.609679    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-794cct6cj"
Nov 01 05:35:53.663989 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:35:53.663958    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a89bfe1-3a8a-4fd6-9b3b-d02e3c0695dc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-794cct6cj\" (UID: \"2a89bfe1-3a8a-4fd6-9b3b-d02e3c0695dc\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-794cct6cj"
Nov 01 05:35:53.664139 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:35:53.664106    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2a89bfe1-3a8a-4fd6-9b3b-d02e3c0695dc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-794cct6cj\" (UID: \"2a89bfe1-3a8a-4fd6-9b3b-d02e3c0695dc\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-794cct6cj"
Nov 01 05:35:53.664252 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:35:53.664217    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp8w9\" (UniqueName: \"kubernetes.io/projected/2a89bfe1-3a8a-4fd6-9b3b-d02e3c0695dc-kube-api-access-cp8w9\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-794cct6cj\" (UID: \"2a89bfe1-3a8a-4fd6-9b3b-d02e3c0695dc\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-794cct6cj"
Nov 01 05:35:53.664315 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:35:53.664297    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2a89bfe1-3a8a-4fd6-9b3b-d02e3c0695dc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-794cct6cj\" (UID: \"2a89bfe1-3a8a-4fd6-9b3b-d02e3c0695dc\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-794cct6cj"
Nov 01 05:35:53.664745 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:35:53.664722    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c6e99c\"/\"routing-overlay\""
Nov 01 05:35:53.692743 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:35:53.692721    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c6e99c\"/\"payload-processing-external-model-plugins-e2e-extmod-b-c6e99c\""
Nov 01 05:35:53.714143 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:35:53.714122    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-c6e99c\"/\"agc-cred-17c481ce95fa\""
Nov 01 05:35:53.714259 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:35:53.714168    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-c6e99c\"/\"agc-cred-3e39090bc554\""
Nov 01 05:35:53.714259 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:35:53.714178    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-c6e99c\"/\"payload-processing-external-model-e2e--8ea5e171-dockercfg-hr8xq\""
Nov 01 05:35:53.714259 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:35:53.714212    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-c6e99c\"/\"agc-cred-8eb61444db22\""
Nov 01 05:35:53.716607 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:35:53.716587    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-794cct6cj"]
Nov 01 05:35:53.765725 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:35:53.765700    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp8w9\" (UniqueName: \"kubernetes.io/projected/2a89bfe1-3a8a-4fd6-9b3b-d02e3c0695dc-kube-api-access-cp8w9\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-794cct6cj\" (UID: \"2a89bfe1-3a8a-4fd6-9b3b-d02e3c0695dc\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-794cct6cj"
Nov 01 05:35:53.765840 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:35:53.765806    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2a89bfe1-3a8a-4fd6-9b3b-d02e3c0695dc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-794cct6cj\" (UID: \"2a89bfe1-3a8a-4fd6-9b3b-d02e3c0695dc\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-794cct6cj"
Nov 01 05:35:53.765928 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:35:53.765914    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a89bfe1-3a8a-4fd6-9b3b-d02e3c0695dc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-794cct6cj\" (UID: \"2a89bfe1-3a8a-4fd6-9b3b-d02e3c0695dc\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-794cct6cj"
Nov 01 05:35:53.766033 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:35:53.766008    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2a89bfe1-3a8a-4fd6-9b3b-d02e3c0695dc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-794cct6cj\" (UID: \"2a89bfe1-3a8a-4fd6-9b3b-d02e3c0695dc\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-794cct6cj"
Nov 01 05:35:53.766506 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:35:53.766485    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2a89bfe1-3a8a-4fd6-9b3b-d02e3c0695dc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-794cct6cj\" (UID: \"2a89bfe1-3a8a-4fd6-9b3b-d02e3c0695dc\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-794cct6cj"
Nov 01 05:35:53.766770 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:35:53.766750    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a89bfe1-3a8a-4fd6-9b3b-d02e3c0695dc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-794cct6cj\" (UID: \"2a89bfe1-3a8a-4fd6-9b3b-d02e3c0695dc\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-794cct6cj"
Nov 01 05:35:53.768719 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:35:53.768702    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2a89bfe1-3a8a-4fd6-9b3b-d02e3c0695dc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-794cct6cj\" (UID: \"2a89bfe1-3a8a-4fd6-9b3b-d02e3c0695dc\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-794cct6cj"
Nov 01 05:35:53.780907 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:35:53.780890    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c6e99c\"/\"kube-root-ca.crt\""
Nov 01 05:35:53.781345 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:35:53.781329    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c6e99c\"/\"openshift-service-ca.crt\""
Nov 01 05:35:53.866394 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:35:53.866332    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp8w9\" (UniqueName: \"kubernetes.io/projected/2a89bfe1-3a8a-4fd6-9b3b-d02e3c0695dc-kube-api-access-cp8w9\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-794cct6cj\" (UID: \"2a89bfe1-3a8a-4fd6-9b3b-d02e3c0695dc\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-794cct6cj"
Nov 01 05:35:53.927985 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:35:53.927964    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-794cct6cj"
Nov 01 05:35:53.977484 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:35:53.977292    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:35:54.346324 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:35:54.346300    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-794cct6cj"]
Nov 01 05:35:54.348700 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:35:54.348671    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a89bfe1_3a8a_4fd6_9b3b_d02e3c0695dc.slice/crio-7151c87f237672253fdc59f1159eb5a92562898a8afaf12b487c83c0dda48854 WatchSource:0}: Error finding container 7151c87f237672253fdc59f1159eb5a92562898a8afaf12b487c83c0dda48854: Status 404 returned error can't find the container with id 7151c87f237672253fdc59f1159eb5a92562898a8afaf12b487c83c0dda48854
Nov 01 05:35:55.055203 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:35:55.055172    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-794cct6cj" event={"ID":"2a89bfe1-3a8a-4fd6-9b3b-d02e3c0695dc","Type":"ContainerStarted","Data":"c2fdcadbfb4467729e2f10f93d284bb4b0f285ed55e30396906ba2efe7bf815f"}
Nov 01 05:35:55.055654 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:35:55.055228    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-794cct6cj" event={"ID":"2a89bfe1-3a8a-4fd6-9b3b-d02e3c0695dc","Type":"ContainerStarted","Data":"7151c87f237672253fdc59f1159eb5a92562898a8afaf12b487c83c0dda48854"}
Nov 01 05:35:55.055654 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:35:55.055258    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-794cct6cj"
Nov 01 05:35:55.171930 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:35:55.171875    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-794cct6cj" podStartSLOduration=2.171858092 podStartE2EDuration="2.171858092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:35:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:35:55.171035328 +0000 UTC m=+1933.241832044" watchObservedRunningTime="2026-11-01 05:35:55.171858092 +0000 UTC m=+1933.242654814"
Nov 01 05:35:58.725057 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:35:58.725026    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:36:00.069607 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:36:00.069575    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:36:01.278016 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:01.277985    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:36:01.475314 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:01.475269    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:36:03.509385 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:03.509291    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:36:04.631370 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:04.631317    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:36:06.021769 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:06.021738    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c2zj8"]
Nov 01 05:36:06.025633 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:06.025615    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c2zj8"
Nov 01 05:36:06.069156 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:06.069133    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-794cct6cj"
Nov 01 05:36:06.100609 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:06.100586    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da1ad4a4-629e-4ce5-a336-1bf832074db4-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-c2zj8\" (UID: \"da1ad4a4-629e-4ce5-a336-1bf832074db4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c2zj8"
Nov 01 05:36:06.100732 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:06.100631    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da1ad4a4-629e-4ce5-a336-1bf832074db4-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-c2zj8\" (UID: \"da1ad4a4-629e-4ce5-a336-1bf832074db4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c2zj8"
Nov 01 05:36:06.100781 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:06.100763    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da1ad4a4-629e-4ce5-a336-1bf832074db4-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-c2zj8\" (UID: \"da1ad4a4-629e-4ce5-a336-1bf832074db4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c2zj8"
Nov 01 05:36:06.100817 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:06.100805    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da1ad4a4-629e-4ce5-a336-1bf832074db4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-c2zj8\" (UID: \"da1ad4a4-629e-4ce5-a336-1bf832074db4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c2zj8"
Nov 01 05:36:06.100898 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:06.100882    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da1ad4a4-629e-4ce5-a336-1bf832074db4-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-c2zj8\" (UID: \"da1ad4a4-629e-4ce5-a336-1bf832074db4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c2zj8"
Nov 01 05:36:06.100932 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:06.100912    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvfx9\" (UniqueName: \"kubernetes.io/projected/da1ad4a4-629e-4ce5-a336-1bf832074db4-kube-api-access-mvfx9\") pod \"maas-default-gateway-openshift-default-58d898bd55-c2zj8\" (UID: \"da1ad4a4-629e-4ce5-a336-1bf832074db4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c2zj8"
Nov 01 05:36:06.100971 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:06.100960    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da1ad4a4-629e-4ce5-a336-1bf832074db4-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-c2zj8\" (UID: \"da1ad4a4-629e-4ce5-a336-1bf832074db4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c2zj8"
Nov 01 05:36:06.101269 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:06.101247    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da1ad4a4-629e-4ce5-a336-1bf832074db4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-c2zj8\" (UID: \"da1ad4a4-629e-4ce5-a336-1bf832074db4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c2zj8"
Nov 01 05:36:06.101367 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:06.101356    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da1ad4a4-629e-4ce5-a336-1bf832074db4-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-c2zj8\" (UID: \"da1ad4a4-629e-4ce5-a336-1bf832074db4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c2zj8"
Nov 01 05:36:06.129684 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:06.129655    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c2zj8"]
Nov 01 05:36:06.202918 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:06.202881    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da1ad4a4-629e-4ce5-a336-1bf832074db4-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-c2zj8\" (UID: \"da1ad4a4-629e-4ce5-a336-1bf832074db4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c2zj8"
Nov 01 05:36:06.203096 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:06.202972    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da1ad4a4-629e-4ce5-a336-1bf832074db4-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-c2zj8\" (UID: \"da1ad4a4-629e-4ce5-a336-1bf832074db4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c2zj8"
Nov 01 05:36:06.203096 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:06.203016    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da1ad4a4-629e-4ce5-a336-1bf832074db4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-c2zj8\" (UID: \"da1ad4a4-629e-4ce5-a336-1bf832074db4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c2zj8"
Nov 01 05:36:06.203096 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:06.203057    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da1ad4a4-629e-4ce5-a336-1bf832074db4-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-c2zj8\" (UID: \"da1ad4a4-629e-4ce5-a336-1bf832074db4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c2zj8"
Nov 01 05:36:06.203096 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:06.203083    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvfx9\" (UniqueName: \"kubernetes.io/projected/da1ad4a4-629e-4ce5-a336-1bf832074db4-kube-api-access-mvfx9\") pod \"maas-default-gateway-openshift-default-58d898bd55-c2zj8\" (UID: \"da1ad4a4-629e-4ce5-a336-1bf832074db4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c2zj8"
Nov 01 05:36:06.203311 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:06.203127    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da1ad4a4-629e-4ce5-a336-1bf832074db4-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-c2zj8\" (UID: \"da1ad4a4-629e-4ce5-a336-1bf832074db4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c2zj8"
Nov 01 05:36:06.203311 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:06.203204    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da1ad4a4-629e-4ce5-a336-1bf832074db4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-c2zj8\" (UID: \"da1ad4a4-629e-4ce5-a336-1bf832074db4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c2zj8"
Nov 01 05:36:06.203311 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:06.203275    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da1ad4a4-629e-4ce5-a336-1bf832074db4-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-c2zj8\" (UID: \"da1ad4a4-629e-4ce5-a336-1bf832074db4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c2zj8"
Nov 01 05:36:06.203503 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:06.203327    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da1ad4a4-629e-4ce5-a336-1bf832074db4-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-c2zj8\" (UID: \"da1ad4a4-629e-4ce5-a336-1bf832074db4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c2zj8"
Nov 01 05:36:06.203503 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:06.203328    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da1ad4a4-629e-4ce5-a336-1bf832074db4-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-c2zj8\" (UID: \"da1ad4a4-629e-4ce5-a336-1bf832074db4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c2zj8"
Nov 01 05:36:06.203625 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:06.203519    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da1ad4a4-629e-4ce5-a336-1bf832074db4-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-c2zj8\" (UID: \"da1ad4a4-629e-4ce5-a336-1bf832074db4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c2zj8"
Nov 01 05:36:06.203731 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:06.203704    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da1ad4a4-629e-4ce5-a336-1bf832074db4-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-c2zj8\" (UID: \"da1ad4a4-629e-4ce5-a336-1bf832074db4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c2zj8"
Nov 01 05:36:06.203837 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:06.203728    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da1ad4a4-629e-4ce5-a336-1bf832074db4-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-c2zj8\" (UID: \"da1ad4a4-629e-4ce5-a336-1bf832074db4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c2zj8"
Nov 01 05:36:06.203837 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:06.203809    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da1ad4a4-629e-4ce5-a336-1bf832074db4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-c2zj8\" (UID: \"da1ad4a4-629e-4ce5-a336-1bf832074db4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c2zj8"
Nov 01 05:36:06.205743 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:06.205723    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da1ad4a4-629e-4ce5-a336-1bf832074db4-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-c2zj8\" (UID: \"da1ad4a4-629e-4ce5-a336-1bf832074db4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c2zj8"
Nov 01 05:36:06.205869 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:06.205852    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da1ad4a4-629e-4ce5-a336-1bf832074db4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-c2zj8\" (UID: \"da1ad4a4-629e-4ce5-a336-1bf832074db4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c2zj8"
Nov 01 05:36:06.266037 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:06.266012    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da1ad4a4-629e-4ce5-a336-1bf832074db4-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-c2zj8\" (UID: \"da1ad4a4-629e-4ce5-a336-1bf832074db4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c2zj8"
Nov 01 05:36:06.266268 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:06.266247    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvfx9\" (UniqueName: \"kubernetes.io/projected/da1ad4a4-629e-4ce5-a336-1bf832074db4-kube-api-access-mvfx9\") pod \"maas-default-gateway-openshift-default-58d898bd55-c2zj8\" (UID: \"da1ad4a4-629e-4ce5-a336-1bf832074db4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c2zj8"
Nov 01 05:36:06.339895 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:06.339821    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c2zj8"
Nov 01 05:36:06.769324 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:06.769301    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c2zj8"]
Nov 01 05:36:06.771904 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:36:06.771881    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda1ad4a4_629e_4ce5_a336_1bf832074db4.slice/crio-da105a73fe8677a10ae7bfb71fa1dde0e47bb089f4050bde43935bf961e84dd7 WatchSource:0}: Error finding container da105a73fe8677a10ae7bfb71fa1dde0e47bb089f4050bde43935bf961e84dd7: Status 404 returned error can't find the container with id da105a73fe8677a10ae7bfb71fa1dde0e47bb089f4050bde43935bf961e84dd7
Nov 01 05:36:06.774269 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:06.774236    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:36:06.774393 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:06.774311    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:36:06.774393 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:06.774357    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:36:07.134167 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:07.134033    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c2zj8" event={"ID":"da1ad4a4-629e-4ce5-a336-1bf832074db4","Type":"ContainerStarted","Data":"089b3a823924d2a44720451f9f96d5a406e59222190c02ab126a83d3230504a9"}
Nov 01 05:36:07.134167 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:07.134099    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c2zj8" event={"ID":"da1ad4a4-629e-4ce5-a336-1bf832074db4","Type":"ContainerStarted","Data":"da105a73fe8677a10ae7bfb71fa1dde0e47bb089f4050bde43935bf961e84dd7"}
Nov 01 05:36:07.255740 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:07.255688    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c2zj8" podStartSLOduration=2.25567427 podStartE2EDuration="2.25567427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:36:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:36:07.253111471 +0000 UTC m=+1945.323908192" watchObservedRunningTime="2026-11-01 05:36:07.25567427 +0000 UTC m=+1945.326471050"
Nov 01 05:36:07.340665 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:07.340628    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c2zj8"
Nov 01 05:36:07.342555 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:07.342526    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-c2zj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 01 05:36:07.342681 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:07.342589    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c2zj8" podUID="da1ad4a4-629e-4ce5-a336-1bf832074db4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 01 05:36:07.932555 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:07.932510    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:36:08.116154 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:08.116110    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:36:08.340477 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:08.340363    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-c2zj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 01 05:36:08.340477 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:08.340434    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c2zj8" podUID="da1ad4a4-629e-4ce5-a336-1bf832074db4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 01 05:36:08.499085 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:08.499049    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:36:09.340418 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:09.340385    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-c2zj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 01 05:36:09.340610 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:09.340464    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c2zj8" podUID="da1ad4a4-629e-4ce5-a336-1bf832074db4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 01 05:36:10.340252 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:10.340214    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-c2zj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 01 05:36:10.340417 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:10.340274    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c2zj8" podUID="da1ad4a4-629e-4ce5-a336-1bf832074db4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 01 05:36:11.048873 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:11.048831    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:36:11.238972 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:11.238921    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:36:11.340766 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:11.340683    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-c2zj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 01 05:36:11.340766 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:11.340747    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c2zj8" podUID="da1ad4a4-629e-4ce5-a336-1bf832074db4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 01 05:36:12.340210 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:12.340173    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-c2zj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 01 05:36:12.340709 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:12.340229    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c2zj8" podUID="da1ad4a4-629e-4ce5-a336-1bf832074db4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 01 05:36:13.340833 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:13.340796    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-c2zj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 01 05:36:13.341286 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:13.340858    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c2zj8" podUID="da1ad4a4-629e-4ce5-a336-1bf832074db4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 01 05:36:13.487415 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:13.487384    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:36:13.679051 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:13.678954    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:36:13.869475 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:13.869419    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:36:14.340892 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:14.340859    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-c2zj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 01 05:36:14.341285 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:14.340909    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c2zj8" podUID="da1ad4a4-629e-4ce5-a336-1bf832074db4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 01 05:36:15.144475 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:15.144112    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:36:15.487897 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:15.487873    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c2zj8"
Nov 01 05:36:15.489357 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:15.489244    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c2zj8"
Nov 01 05:36:15.492716 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:15.492674    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c2zj8"
Nov 01 05:36:18.458467 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:18.458380    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:36:23.319101 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:23.319062    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:36:27.753939 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:27.753897    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:36:28.221871 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:28.221834    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:36:30.069422 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:36:30.069392    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:36:33.031700 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:33.031664    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:36:36.236048 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:36.236013    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vwd6x"]
Nov 01 05:36:36.240012 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:36.239992    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vwd6x"
Nov 01 05:36:36.279370 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:36.279329    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b840b67-ec8d-45e0-91b4-0a8dc7a353eb-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-vwd6x\" (UID: \"5b840b67-ec8d-45e0-91b4-0a8dc7a353eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vwd6x"
Nov 01 05:36:36.279567 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:36.279428    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b840b67-ec8d-45e0-91b4-0a8dc7a353eb-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-vwd6x\" (UID: \"5b840b67-ec8d-45e0-91b4-0a8dc7a353eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vwd6x"
Nov 01 05:36:36.279567 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:36.279498    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b840b67-ec8d-45e0-91b4-0a8dc7a353eb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-vwd6x\" (UID: \"5b840b67-ec8d-45e0-91b4-0a8dc7a353eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vwd6x"
Nov 01 05:36:36.279567 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:36.279525    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b840b67-ec8d-45e0-91b4-0a8dc7a353eb-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-vwd6x\" (UID: \"5b840b67-ec8d-45e0-91b4-0a8dc7a353eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vwd6x"
Nov 01 05:36:36.279727 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:36.279676    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b840b67-ec8d-45e0-91b4-0a8dc7a353eb-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-vwd6x\" (UID: \"5b840b67-ec8d-45e0-91b4-0a8dc7a353eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vwd6x"
Nov 01 05:36:36.279727 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:36.279701    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k27zz\" (UniqueName: \"kubernetes.io/projected/5b840b67-ec8d-45e0-91b4-0a8dc7a353eb-kube-api-access-k27zz\") pod \"maas-default-gateway-openshift-default-58d898bd55-vwd6x\" (UID: \"5b840b67-ec8d-45e0-91b4-0a8dc7a353eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vwd6x"
Nov 01 05:36:36.279827 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:36.279809    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b840b67-ec8d-45e0-91b4-0a8dc7a353eb-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-vwd6x\" (UID: \"5b840b67-ec8d-45e0-91b4-0a8dc7a353eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vwd6x"
Nov 01 05:36:36.279873 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:36.279862    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b840b67-ec8d-45e0-91b4-0a8dc7a353eb-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-vwd6x\" (UID: \"5b840b67-ec8d-45e0-91b4-0a8dc7a353eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vwd6x"
Nov 01 05:36:36.279959 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:36.279940    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b840b67-ec8d-45e0-91b4-0a8dc7a353eb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-vwd6x\" (UID: \"5b840b67-ec8d-45e0-91b4-0a8dc7a353eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vwd6x"
Nov 01 05:36:36.345749 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:36.345714    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vwd6x"]
Nov 01 05:36:36.381131 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:36.381102    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b840b67-ec8d-45e0-91b4-0a8dc7a353eb-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-vwd6x\" (UID: \"5b840b67-ec8d-45e0-91b4-0a8dc7a353eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vwd6x"
Nov 01 05:36:36.381346 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:36.381173    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b840b67-ec8d-45e0-91b4-0a8dc7a353eb-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-vwd6x\" (UID: \"5b840b67-ec8d-45e0-91b4-0a8dc7a353eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vwd6x"
Nov 01 05:36:36.381346 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:36.381231    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b840b67-ec8d-45e0-91b4-0a8dc7a353eb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-vwd6x\" (UID: \"5b840b67-ec8d-45e0-91b4-0a8dc7a353eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vwd6x"
Nov 01 05:36:36.381346 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:36.381282    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b840b67-ec8d-45e0-91b4-0a8dc7a353eb-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-vwd6x\" (UID: \"5b840b67-ec8d-45e0-91b4-0a8dc7a353eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vwd6x"
Nov 01 05:36:36.381346 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:36.381315    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b840b67-ec8d-45e0-91b4-0a8dc7a353eb-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-vwd6x\" (UID: \"5b840b67-ec8d-45e0-91b4-0a8dc7a353eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vwd6x"
Nov 01 05:36:36.381554 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:36.381359    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b840b67-ec8d-45e0-91b4-0a8dc7a353eb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-vwd6x\" (UID: \"5b840b67-ec8d-45e0-91b4-0a8dc7a353eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vwd6x"
Nov 01 05:36:36.381554 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:36.381398    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b840b67-ec8d-45e0-91b4-0a8dc7a353eb-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-vwd6x\" (UID: \"5b840b67-ec8d-45e0-91b4-0a8dc7a353eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vwd6x"
Nov 01 05:36:36.381554 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:36.381492    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b840b67-ec8d-45e0-91b4-0a8dc7a353eb-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-vwd6x\" (UID: \"5b840b67-ec8d-45e0-91b4-0a8dc7a353eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vwd6x"
Nov 01 05:36:36.381554 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:36.381527    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k27zz\" (UniqueName: \"kubernetes.io/projected/5b840b67-ec8d-45e0-91b4-0a8dc7a353eb-kube-api-access-k27zz\") pod \"maas-default-gateway-openshift-default-58d898bd55-vwd6x\" (UID: \"5b840b67-ec8d-45e0-91b4-0a8dc7a353eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vwd6x"
Nov 01 05:36:36.381728 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:36.381612    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b840b67-ec8d-45e0-91b4-0a8dc7a353eb-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-vwd6x\" (UID: \"5b840b67-ec8d-45e0-91b4-0a8dc7a353eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vwd6x"
Nov 01 05:36:36.381890 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:36.381866    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b840b67-ec8d-45e0-91b4-0a8dc7a353eb-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-vwd6x\" (UID: \"5b840b67-ec8d-45e0-91b4-0a8dc7a353eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vwd6x"
Nov 01 05:36:36.382056 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:36.382032    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b840b67-ec8d-45e0-91b4-0a8dc7a353eb-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-vwd6x\" (UID: \"5b840b67-ec8d-45e0-91b4-0a8dc7a353eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vwd6x"
Nov 01 05:36:36.382592 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:36.382564    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b840b67-ec8d-45e0-91b4-0a8dc7a353eb-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-vwd6x\" (UID: \"5b840b67-ec8d-45e0-91b4-0a8dc7a353eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vwd6x"
Nov 01 05:36:36.382693 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:36.382660    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b840b67-ec8d-45e0-91b4-0a8dc7a353eb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-vwd6x\" (UID: \"5b840b67-ec8d-45e0-91b4-0a8dc7a353eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vwd6x"
Nov 01 05:36:36.383965 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:36.383942    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b840b67-ec8d-45e0-91b4-0a8dc7a353eb-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-vwd6x\" (UID: \"5b840b67-ec8d-45e0-91b4-0a8dc7a353eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vwd6x"
Nov 01 05:36:36.384911 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:36.384893    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b840b67-ec8d-45e0-91b4-0a8dc7a353eb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-vwd6x\" (UID: \"5b840b67-ec8d-45e0-91b4-0a8dc7a353eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vwd6x"
Nov 01 05:36:36.442222 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:36.442189    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b840b67-ec8d-45e0-91b4-0a8dc7a353eb-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-vwd6x\" (UID: \"5b840b67-ec8d-45e0-91b4-0a8dc7a353eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vwd6x"
Nov 01 05:36:36.442354 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:36.442302    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k27zz\" (UniqueName: \"kubernetes.io/projected/5b840b67-ec8d-45e0-91b4-0a8dc7a353eb-kube-api-access-k27zz\") pod \"maas-default-gateway-openshift-default-58d898bd55-vwd6x\" (UID: \"5b840b67-ec8d-45e0-91b4-0a8dc7a353eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vwd6x"
Nov 01 05:36:36.529977 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:36.529899    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-794cct6cj"]
Nov 01 05:36:36.555563 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:36.555536    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vwd6x"
Nov 01 05:36:36.986903 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:36.986878    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vwd6x"]
Nov 01 05:36:36.989149 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:36:36.989121    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b840b67_ec8d_45e0_91b4_0a8dc7a353eb.slice/crio-42f0e3d432167355a397d4fe650a3bdf47ea99a21279b6fe0e4b88aa0f9afcaf WatchSource:0}: Error finding container 42f0e3d432167355a397d4fe650a3bdf47ea99a21279b6fe0e4b88aa0f9afcaf: Status 404 returned error can't find the container with id 42f0e3d432167355a397d4fe650a3bdf47ea99a21279b6fe0e4b88aa0f9afcaf
Nov 01 05:36:36.991669 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:36.991634    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:36:36.991782 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:36.991715    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:36:36.991782 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:36.991763    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:36:37.328715 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:37.328622    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vwd6x" event={"ID":"5b840b67-ec8d-45e0-91b4-0a8dc7a353eb","Type":"ContainerStarted","Data":"2b5c9cf467da138c68971f4e06576313f60d33f678dad045b4a19f60e06d260a"}
Nov 01 05:36:37.328715 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:37.328682    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vwd6x" event={"ID":"5b840b67-ec8d-45e0-91b4-0a8dc7a353eb","Type":"ContainerStarted","Data":"42f0e3d432167355a397d4fe650a3bdf47ea99a21279b6fe0e4b88aa0f9afcaf"}
Nov 01 05:36:37.452530 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:37.451100    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vwd6x" podStartSLOduration=1.451081148 podStartE2EDuration="1.451081148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:36:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:36:37.450295275 +0000 UTC m=+1975.521092037" watchObservedRunningTime="2026-11-01 05:36:37.451081148 +0000 UTC m=+1975.521877870"
Nov 01 05:36:37.555672 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:37.555635    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vwd6x"
Nov 01 05:36:37.905256 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:37.905189    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:36:38.558084 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:38.558053    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-vwd6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Nov 01 05:36:38.558469 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:38.558110    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vwd6x" podUID="5b840b67-ec8d-45e0-91b4-0a8dc7a353eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Nov 01 05:36:39.555978 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:39.555943    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-vwd6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Nov 01 05:36:39.556160 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:39.555997    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vwd6x" podUID="5b840b67-ec8d-45e0-91b4-0a8dc7a353eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Nov 01 05:36:40.556823 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:40.556784    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-vwd6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Nov 01 05:36:40.557192 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:40.556839    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vwd6x" podUID="5b840b67-ec8d-45e0-91b4-0a8dc7a353eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Nov 01 05:36:41.530669 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:41.530628    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-794cct6cj" podUID="2a89bfe1-3a8a-4fd6-9b3b-d02e3c0695dc" containerName="payload-processing" containerID="cri-o://c2fdcadbfb4467729e2f10f93d284bb4b0f285ed55e30396906ba2efe7bf815f" gracePeriod=25
Nov 01 05:36:41.556537 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:41.556511    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-vwd6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Nov 01 05:36:41.556620 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:41.556575    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vwd6x" podUID="5b840b67-ec8d-45e0-91b4-0a8dc7a353eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Nov 01 05:36:41.843023 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:41.843000    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-794cct6cj"
Nov 01 05:36:41.963811 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:41.963023    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a89bfe1-3a8a-4fd6-9b3b-d02e3c0695dc-kube-api-access-cp8w9\" (UniqueName: \"kubernetes.io/projected/2a89bfe1-3a8a-4fd6-9b3b-d02e3c0695dc-kube-api-access-cp8w9\") pod \"2a89bfe1-3a8a-4fd6-9b3b-d02e3c0695dc\" (UID: \"2a89bfe1-3a8a-4fd6-9b3b-d02e3c0695dc\") "
Nov 01 05:36:41.963811 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:41.963099    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a89bfe1-3a8a-4fd6-9b3b-d02e3c0695dc-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2a89bfe1-3a8a-4fd6-9b3b-d02e3c0695dc-routing-overlay-volume\") pod \"2a89bfe1-3a8a-4fd6-9b3b-d02e3c0695dc\" (UID: \"2a89bfe1-3a8a-4fd6-9b3b-d02e3c0695dc\") "
Nov 01 05:36:41.963811 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:41.963152    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a89bfe1-3a8a-4fd6-9b3b-d02e3c0695dc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a89bfe1-3a8a-4fd6-9b3b-d02e3c0695dc-plugins-config-volume\") pod \"2a89bfe1-3a8a-4fd6-9b3b-d02e3c0695dc\" (UID: \"2a89bfe1-3a8a-4fd6-9b3b-d02e3c0695dc\") "
Nov 01 05:36:41.963811 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:41.963338    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a89bfe1-3a8a-4fd6-9b3b-d02e3c0695dc-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2a89bfe1-3a8a-4fd6-9b3b-d02e3c0695dc-provider-credentials\") pod \"2a89bfe1-3a8a-4fd6-9b3b-d02e3c0695dc\" (UID: \"2a89bfe1-3a8a-4fd6-9b3b-d02e3c0695dc\") "
Nov 01 05:36:41.963811 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:41.963556    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a89bfe1-3a8a-4fd6-9b3b-d02e3c0695dc-routing-overlay-volume" pod "2a89bfe1-3a8a-4fd6-9b3b-d02e3c0695dc" (UID: "2a89bfe1-3a8a-4fd6-9b3b-d02e3c0695dc"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:36:41.963811 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:41.963602    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a89bfe1-3a8a-4fd6-9b3b-d02e3c0695dc-plugins-config-volume" pod "2a89bfe1-3a8a-4fd6-9b3b-d02e3c0695dc" (UID: "2a89bfe1-3a8a-4fd6-9b3b-d02e3c0695dc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:36:41.964191 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:41.963847    5689 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2a89bfe1-3a8a-4fd6-9b3b-d02e3c0695dc-routing-overlay-volume\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:36:41.964191 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:41.963865    5689 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a89bfe1-3a8a-4fd6-9b3b-d02e3c0695dc-plugins-config-volume\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:36:41.966552 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:41.966504    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a89bfe1-3a8a-4fd6-9b3b-d02e3c0695dc-kube-api-access-cp8w9" pod "2a89bfe1-3a8a-4fd6-9b3b-d02e3c0695dc" (UID: "2a89bfe1-3a8a-4fd6-9b3b-d02e3c0695dc"). InnerVolumeSpecName "kube-api-access-cp8w9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:36:41.968037 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:41.968009    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a89bfe1-3a8a-4fd6-9b3b-d02e3c0695dc-provider-credentials" pod "2a89bfe1-3a8a-4fd6-9b3b-d02e3c0695dc" (UID: "2a89bfe1-3a8a-4fd6-9b3b-d02e3c0695dc"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:36:42.105647 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:42.105566    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cp8w9\" (UniqueName: \"kubernetes.io/projected/2a89bfe1-3a8a-4fd6-9b3b-d02e3c0695dc-kube-api-access-cp8w9\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:36:42.105647 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:42.105595    5689 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2a89bfe1-3a8a-4fd6-9b3b-d02e3c0695dc-provider-credentials\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:36:42.357880 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:42.357797    5689 generic.go:356] "Generic (PLEG): container finished" podID="2a89bfe1-3a8a-4fd6-9b3b-d02e3c0695dc" containerID="c2fdcadbfb4467729e2f10f93d284bb4b0f285ed55e30396906ba2efe7bf815f" exitCode=0
Nov 01 05:36:42.357880 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:42.357855    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-794cct6cj"
Nov 01 05:36:42.358067 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:42.357883    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-794cct6cj" event={"ID":"2a89bfe1-3a8a-4fd6-9b3b-d02e3c0695dc","Type":"ContainerDied","Data":"c2fdcadbfb4467729e2f10f93d284bb4b0f285ed55e30396906ba2efe7bf815f"}
Nov 01 05:36:42.358067 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:42.357939    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-794cct6cj" event={"ID":"2a89bfe1-3a8a-4fd6-9b3b-d02e3c0695dc","Type":"ContainerDied","Data":"7151c87f237672253fdc59f1159eb5a92562898a8afaf12b487c83c0dda48854"}
Nov 01 05:36:42.358067 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:42.357960    5689 scope.go:122] "RemoveContainer" containerID="c2fdcadbfb4467729e2f10f93d284bb4b0f285ed55e30396906ba2efe7bf815f"
Nov 01 05:36:42.367199 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:42.367182    5689 scope.go:122] "RemoveContainer" containerID="c2fdcadbfb4467729e2f10f93d284bb4b0f285ed55e30396906ba2efe7bf815f"
Nov 01 05:36:42.367468 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:36:42.367431    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2fdcadbfb4467729e2f10f93d284bb4b0f285ed55e30396906ba2efe7bf815f\": container with ID starting with c2fdcadbfb4467729e2f10f93d284bb4b0f285ed55e30396906ba2efe7bf815f not found: ID does not exist" containerID="c2fdcadbfb4467729e2f10f93d284bb4b0f285ed55e30396906ba2efe7bf815f"
Nov 01 05:36:42.367563 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:42.367475    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2fdcadbfb4467729e2f10f93d284bb4b0f285ed55e30396906ba2efe7bf815f"} err="failed to get container status \"c2fdcadbfb4467729e2f10f93d284bb4b0f285ed55e30396906ba2efe7bf815f\": rpc error: code = NotFound desc = could not find container \"c2fdcadbfb4467729e2f10f93d284bb4b0f285ed55e30396906ba2efe7bf815f\": container with ID starting with c2fdcadbfb4467729e2f10f93d284bb4b0f285ed55e30396906ba2efe7bf815f not found: ID does not exist"
Nov 01 05:36:42.522396 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:42.522365    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-794cct6cj"]
Nov 01 05:36:42.555826 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:42.555796    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-vwd6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Nov 01 05:36:42.555985 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:42.555864    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vwd6x" podUID="5b840b67-ec8d-45e0-91b4-0a8dc7a353eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Nov 01 05:36:42.571479 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:42.571440    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-794cct6cj"]
Nov 01 05:36:42.752599 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:42.752569    5689 scope.go:122] "RemoveContainer" containerID="412a661093710710b6e4b51c65196912cadaee5a187ebe7551256efb35cfe0ec"
Nov 01 05:36:42.852225 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:42.852194    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:36:43.556191 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:43.556164    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-vwd6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Nov 01 05:36:43.556477 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:43.556210    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vwd6x" podUID="5b840b67-ec8d-45e0-91b4-0a8dc7a353eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Nov 01 05:36:44.068312 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:44.068282    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a89bfe1-3a8a-4fd6-9b3b-d02e3c0695dc" path="/var/lib/kubelet/pods/2a89bfe1-3a8a-4fd6-9b3b-d02e3c0695dc/volumes"
Nov 01 05:36:44.438001 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:44.437969    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:36:44.555986 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:44.555949    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-vwd6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Nov 01 05:36:44.556150 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:44.556012    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vwd6x" podUID="5b840b67-ec8d-45e0-91b4-0a8dc7a353eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Nov 01 05:36:44.616127 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:44.616093    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:36:45.560496 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:45.560464    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vwd6x"
Nov 01 05:36:45.560901 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:45.560869    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vwd6x"
Nov 01 05:36:45.561621 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:45.561599    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vwd6x"
Nov 01 05:36:47.727158 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:47.727108    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:36:52.541192 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:52.541109    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:36:53.801051 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:53.801019    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:36:57.491276 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:36:57.491244    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:37:00.072146 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:37:00.072106    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:37:02.323905 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:02.323873    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:37:04.974790 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:04.974759    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:37:07.123600 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:07.123553    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:37:11.701428 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:11.701391    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:37:12.072672 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:12.072600    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:37:13.825584 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:13.825554    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:37:16.963152 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:16.963124    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:37:18.141153 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:18.141125    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:37:22.148601 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:22.148569    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:37:24.356951 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:24.356917    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb"]
Nov 01 05:37:24.357496 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:24.357103    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb" podUID="c8a582fb-747c-4c55-a0e3-b5f5139fa28f" containerName="istio-proxy" containerID="cri-o://977315bedf667832fb45730d9da5f32d29e00bd9ffcc867e2b1fef2900eb0767" gracePeriod=30
Nov 01 05:37:24.411744 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:24.411709    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4"]
Nov 01 05:37:24.411955 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:24.411929    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4" podUID="d885f72a-2f29-41ee-b803-ab1a3a96fc32" containerName="istio-proxy" containerID="cri-o://daec85060065e22ccd8f7539888b7f6ec2d75b876fa53258eb3164ff68a3fb6e" gracePeriod=30
Nov 01 05:37:24.413946 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:24.413918    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws"]
Nov 01 05:37:24.414143 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:24.414100    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws" podUID="0eccaa64-f564-44d7-a611-9ff326b899b6" containerName="istio-proxy" containerID="cri-o://ae218ab3dbdf626581a3f783a88ea00cef604ec42e1fa82ccea0faa4ff0b019d" gracePeriod=30
Nov 01 05:37:24.461856 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:24.461475    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl"]
Nov 01 05:37:24.461856 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:24.461695    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl" podUID="4c7f07f0-82ce-4e5b-97c5-946cc901566d" containerName="istio-proxy" containerID="cri-o://a6e7821944ee9614d85fc9b232cf0f63caa1e755014ab9f1eb2029146e0d9415" gracePeriod=30
Nov 01 05:37:26.925297 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:26.925263    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:37:28.614935 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:28.614907    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:37:28.800683 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:28.800646    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:37:29.939395 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:29.935001    5689 generic.go:356] "Generic (PLEG): container finished" podID="d885f72a-2f29-41ee-b803-ab1a3a96fc32" containerID="daec85060065e22ccd8f7539888b7f6ec2d75b876fa53258eb3164ff68a3fb6e" exitCode=0
Nov 01 05:37:29.939395 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:29.935117    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4" event={"ID":"d885f72a-2f29-41ee-b803-ab1a3a96fc32","Type":"ContainerDied","Data":"daec85060065e22ccd8f7539888b7f6ec2d75b876fa53258eb3164ff68a3fb6e"}
Nov 01 05:37:29.956007 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:29.955828    5689 generic.go:356] "Generic (PLEG): container finished" podID="0eccaa64-f564-44d7-a611-9ff326b899b6" containerID="ae218ab3dbdf626581a3f783a88ea00cef604ec42e1fa82ccea0faa4ff0b019d" exitCode=0
Nov 01 05:37:29.956007 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:29.955932    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws" event={"ID":"0eccaa64-f564-44d7-a611-9ff326b899b6","Type":"ContainerDied","Data":"ae218ab3dbdf626581a3f783a88ea00cef604ec42e1fa82ccea0faa4ff0b019d"}
Nov 01 05:37:29.960180 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:29.960099    5689 generic.go:356] "Generic (PLEG): container finished" podID="4c7f07f0-82ce-4e5b-97c5-946cc901566d" containerID="a6e7821944ee9614d85fc9b232cf0f63caa1e755014ab9f1eb2029146e0d9415" exitCode=0
Nov 01 05:37:29.960924 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:29.960900    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl" event={"ID":"4c7f07f0-82ce-4e5b-97c5-946cc901566d","Type":"ContainerDied","Data":"a6e7821944ee9614d85fc9b232cf0f63caa1e755014ab9f1eb2029146e0d9415"}
Nov 01 05:37:29.963220 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:29.963198    5689 generic.go:356] "Generic (PLEG): container finished" podID="c8a582fb-747c-4c55-a0e3-b5f5139fa28f" containerID="977315bedf667832fb45730d9da5f32d29e00bd9ffcc867e2b1fef2900eb0767" exitCode=0
Nov 01 05:37:29.963433 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:29.963417    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb" event={"ID":"c8a582fb-747c-4c55-a0e3-b5f5139fa28f","Type":"ContainerDied","Data":"977315bedf667832fb45730d9da5f32d29e00bd9ffcc867e2b1fef2900eb0767"}
Nov 01 05:37:29.963603 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:29.963588    5689 scope.go:122] "RemoveContainer" containerID="7cadc1665174645d53c05fbe46ab5b2ab55f637c9e8ac49e4e497655859406a2"
Nov 01 05:37:29.965240 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:29.965216    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb"
Nov 01 05:37:30.037479 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.037371    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4"
Nov 01 05:37:30.110489 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.084919    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8a582fb-747c-4c55-a0e3-b5f5139fa28f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8a582fb-747c-4c55-a0e3-b5f5139fa28f-workload-socket\") pod \"c8a582fb-747c-4c55-a0e3-b5f5139fa28f\" (UID: \"c8a582fb-747c-4c55-a0e3-b5f5139fa28f\") "
Nov 01 05:37:30.110489 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.085029    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8a582fb-747c-4c55-a0e3-b5f5139fa28f-kube-api-access-9m9xq\" (UniqueName: \"kubernetes.io/projected/c8a582fb-747c-4c55-a0e3-b5f5139fa28f-kube-api-access-9m9xq\") pod \"c8a582fb-747c-4c55-a0e3-b5f5139fa28f\" (UID: \"c8a582fb-747c-4c55-a0e3-b5f5139fa28f\") "
Nov 01 05:37:30.110489 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.085094    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8a582fb-747c-4c55-a0e3-b5f5139fa28f-istio-token\" (UniqueName: \"kubernetes.io/projected/c8a582fb-747c-4c55-a0e3-b5f5139fa28f-istio-token\") pod \"c8a582fb-747c-4c55-a0e3-b5f5139fa28f\" (UID: \"c8a582fb-747c-4c55-a0e3-b5f5139fa28f\") "
Nov 01 05:37:30.110489 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.085146    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8a582fb-747c-4c55-a0e3-b5f5139fa28f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8a582fb-747c-4c55-a0e3-b5f5139fa28f-istiod-ca-cert\") pod \"c8a582fb-747c-4c55-a0e3-b5f5139fa28f\" (UID: \"c8a582fb-747c-4c55-a0e3-b5f5139fa28f\") "
Nov 01 05:37:30.110489 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.085177    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8a582fb-747c-4c55-a0e3-b5f5139fa28f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8a582fb-747c-4c55-a0e3-b5f5139fa28f-istio-envoy\") pod \"c8a582fb-747c-4c55-a0e3-b5f5139fa28f\" (UID: \"c8a582fb-747c-4c55-a0e3-b5f5139fa28f\") "
Nov 01 05:37:30.110489 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.085233    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8a582fb-747c-4c55-a0e3-b5f5139fa28f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8a582fb-747c-4c55-a0e3-b5f5139fa28f-credential-socket\") pod \"c8a582fb-747c-4c55-a0e3-b5f5139fa28f\" (UID: \"c8a582fb-747c-4c55-a0e3-b5f5139fa28f\") "
Nov 01 05:37:30.110489 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.085266    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c8a582fb-747c-4c55-a0e3-b5f5139fa28f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8a582fb-747c-4c55-a0e3-b5f5139fa28f-istio-podinfo\") pod \"c8a582fb-747c-4c55-a0e3-b5f5139fa28f\" (UID: \"c8a582fb-747c-4c55-a0e3-b5f5139fa28f\") "
Nov 01 05:37:30.110489 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.085392    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8a582fb-747c-4c55-a0e3-b5f5139fa28f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8a582fb-747c-4c55-a0e3-b5f5139fa28f-istio-data\") pod \"c8a582fb-747c-4c55-a0e3-b5f5139fa28f\" (UID: \"c8a582fb-747c-4c55-a0e3-b5f5139fa28f\") "
Nov 01 05:37:30.110489 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.085445    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8a582fb-747c-4c55-a0e3-b5f5139fa28f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8a582fb-747c-4c55-a0e3-b5f5139fa28f-workload-certs\") pod \"c8a582fb-747c-4c55-a0e3-b5f5139fa28f\" (UID: \"c8a582fb-747c-4c55-a0e3-b5f5139fa28f\") "
Nov 01 05:37:30.120477 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.113883    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8a582fb-747c-4c55-a0e3-b5f5139fa28f-workload-socket" pod "c8a582fb-747c-4c55-a0e3-b5f5139fa28f" (UID: "c8a582fb-747c-4c55-a0e3-b5f5139fa28f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:37:30.120477 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.118662    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8a582fb-747c-4c55-a0e3-b5f5139fa28f-istio-data" pod "c8a582fb-747c-4c55-a0e3-b5f5139fa28f" (UID: "c8a582fb-747c-4c55-a0e3-b5f5139fa28f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:37:30.120862 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.120810    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8a582fb-747c-4c55-a0e3-b5f5139fa28f-workload-certs" pod "c8a582fb-747c-4c55-a0e3-b5f5139fa28f" (UID: "c8a582fb-747c-4c55-a0e3-b5f5139fa28f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:37:30.165150 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.158341    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8a582fb-747c-4c55-a0e3-b5f5139fa28f-credential-socket" pod "c8a582fb-747c-4c55-a0e3-b5f5139fa28f" (UID: "c8a582fb-747c-4c55-a0e3-b5f5139fa28f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:37:30.165150 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.158543    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8a582fb-747c-4c55-a0e3-b5f5139fa28f-istiod-ca-cert" pod "c8a582fb-747c-4c55-a0e3-b5f5139fa28f" (UID: "c8a582fb-747c-4c55-a0e3-b5f5139fa28f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:37:30.165150 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:37:30.164540    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:37:30.166370 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.166327    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8a582fb-747c-4c55-a0e3-b5f5139fa28f-kube-api-access-9m9xq" pod "c8a582fb-747c-4c55-a0e3-b5f5139fa28f" (UID: "c8a582fb-747c-4c55-a0e3-b5f5139fa28f"). InnerVolumeSpecName "kube-api-access-9m9xq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:37:30.182949 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.182881    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8a582fb-747c-4c55-a0e3-b5f5139fa28f-istio-token" pod "c8a582fb-747c-4c55-a0e3-b5f5139fa28f" (UID: "c8a582fb-747c-4c55-a0e3-b5f5139fa28f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:37:30.183432 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.183415    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8a582fb-747c-4c55-a0e3-b5f5139fa28f-istio-envoy" pod "c8a582fb-747c-4c55-a0e3-b5f5139fa28f" (UID: "c8a582fb-747c-4c55-a0e3-b5f5139fa28f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:37:30.185643 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.185555    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c8a582fb-747c-4c55-a0e3-b5f5139fa28f-istio-podinfo" pod "c8a582fb-747c-4c55-a0e3-b5f5139fa28f" (UID: "c8a582fb-747c-4c55-a0e3-b5f5139fa28f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 05:37:30.187408 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.186835    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d885f72a-2f29-41ee-b803-ab1a3a96fc32-kube-api-access-jv7pc\" (UniqueName: \"kubernetes.io/projected/d885f72a-2f29-41ee-b803-ab1a3a96fc32-kube-api-access-jv7pc\") pod \"d885f72a-2f29-41ee-b803-ab1a3a96fc32\" (UID: \"d885f72a-2f29-41ee-b803-ab1a3a96fc32\") "
Nov 01 05:37:30.187408 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.186867    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d885f72a-2f29-41ee-b803-ab1a3a96fc32-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d885f72a-2f29-41ee-b803-ab1a3a96fc32-workload-certs\") pod \"d885f72a-2f29-41ee-b803-ab1a3a96fc32\" (UID: \"d885f72a-2f29-41ee-b803-ab1a3a96fc32\") "
Nov 01 05:37:30.187408 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.186952    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d885f72a-2f29-41ee-b803-ab1a3a96fc32-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d885f72a-2f29-41ee-b803-ab1a3a96fc32-istiod-ca-cert\") pod \"d885f72a-2f29-41ee-b803-ab1a3a96fc32\" (UID: \"d885f72a-2f29-41ee-b803-ab1a3a96fc32\") "
Nov 01 05:37:30.187408 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.187037    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d885f72a-2f29-41ee-b803-ab1a3a96fc32-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d885f72a-2f29-41ee-b803-ab1a3a96fc32-istio-data\") pod \"d885f72a-2f29-41ee-b803-ab1a3a96fc32\" (UID: \"d885f72a-2f29-41ee-b803-ab1a3a96fc32\") "
Nov 01 05:37:30.187408 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.187096    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d885f72a-2f29-41ee-b803-ab1a3a96fc32-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d885f72a-2f29-41ee-b803-ab1a3a96fc32-credential-socket\") pod \"d885f72a-2f29-41ee-b803-ab1a3a96fc32\" (UID: \"d885f72a-2f29-41ee-b803-ab1a3a96fc32\") "
Nov 01 05:37:30.187408 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.187135    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d885f72a-2f29-41ee-b803-ab1a3a96fc32-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d885f72a-2f29-41ee-b803-ab1a3a96fc32-istio-envoy\") pod \"d885f72a-2f29-41ee-b803-ab1a3a96fc32\" (UID: \"d885f72a-2f29-41ee-b803-ab1a3a96fc32\") "
Nov 01 05:37:30.187408 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.187174    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d885f72a-2f29-41ee-b803-ab1a3a96fc32-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d885f72a-2f29-41ee-b803-ab1a3a96fc32-istio-podinfo\") pod \"d885f72a-2f29-41ee-b803-ab1a3a96fc32\" (UID: \"d885f72a-2f29-41ee-b803-ab1a3a96fc32\") "
Nov 01 05:37:30.187408 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.187215    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d885f72a-2f29-41ee-b803-ab1a3a96fc32-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d885f72a-2f29-41ee-b803-ab1a3a96fc32-workload-socket\") pod \"d885f72a-2f29-41ee-b803-ab1a3a96fc32\" (UID: \"d885f72a-2f29-41ee-b803-ab1a3a96fc32\") "
Nov 01 05:37:30.187408 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.187298    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d885f72a-2f29-41ee-b803-ab1a3a96fc32-istiod-ca-cert" pod "d885f72a-2f29-41ee-b803-ab1a3a96fc32" (UID: "d885f72a-2f29-41ee-b803-ab1a3a96fc32"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:37:30.187408 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.187317    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d885f72a-2f29-41ee-b803-ab1a3a96fc32-istio-token\" (UniqueName: \"kubernetes.io/projected/d885f72a-2f29-41ee-b803-ab1a3a96fc32-istio-token\") pod \"d885f72a-2f29-41ee-b803-ab1a3a96fc32\" (UID: \"d885f72a-2f29-41ee-b803-ab1a3a96fc32\") "
Nov 01 05:37:30.189883 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.189807    5689 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8a582fb-747c-4c55-a0e3-b5f5139fa28f-istio-data\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:37:30.189883 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.189834    5689 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8a582fb-747c-4c55-a0e3-b5f5139fa28f-workload-certs\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:37:30.189883 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.189848    5689 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8a582fb-747c-4c55-a0e3-b5f5139fa28f-workload-socket\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:37:30.190198 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.189862    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9m9xq\" (UniqueName: \"kubernetes.io/projected/c8a582fb-747c-4c55-a0e3-b5f5139fa28f-kube-api-access-9m9xq\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:37:30.190198 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.190149    5689 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8a582fb-747c-4c55-a0e3-b5f5139fa28f-istio-token\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:37:30.190198 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.190163    5689 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d885f72a-2f29-41ee-b803-ab1a3a96fc32-istiod-ca-cert\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:37:30.190432 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.190175    5689 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8a582fb-747c-4c55-a0e3-b5f5139fa28f-istiod-ca-cert\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:37:30.190599 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.190415    5689 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8a582fb-747c-4c55-a0e3-b5f5139fa28f-istio-envoy\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:37:30.191168 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.190577    5689 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8a582fb-747c-4c55-a0e3-b5f5139fa28f-credential-socket\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:37:30.191168 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.190710    5689 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8a582fb-747c-4c55-a0e3-b5f5139fa28f-istio-podinfo\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:37:30.191168 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.190687    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d885f72a-2f29-41ee-b803-ab1a3a96fc32-istio-data" pod "d885f72a-2f29-41ee-b803-ab1a3a96fc32" (UID: "d885f72a-2f29-41ee-b803-ab1a3a96fc32"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:37:30.191443 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.191410    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d885f72a-2f29-41ee-b803-ab1a3a96fc32-credential-socket" pod "d885f72a-2f29-41ee-b803-ab1a3a96fc32" (UID: "d885f72a-2f29-41ee-b803-ab1a3a96fc32"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:37:30.199024 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.199004    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d885f72a-2f29-41ee-b803-ab1a3a96fc32-workload-certs" pod "d885f72a-2f29-41ee-b803-ab1a3a96fc32" (UID: "d885f72a-2f29-41ee-b803-ab1a3a96fc32"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:37:30.199752 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.199712    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d885f72a-2f29-41ee-b803-ab1a3a96fc32-workload-socket" pod "d885f72a-2f29-41ee-b803-ab1a3a96fc32" (UID: "d885f72a-2f29-41ee-b803-ab1a3a96fc32"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:37:30.204995 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.204906    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl"
Nov 01 05:37:30.206213 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.206178    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d885f72a-2f29-41ee-b803-ab1a3a96fc32-kube-api-access-jv7pc" pod "d885f72a-2f29-41ee-b803-ab1a3a96fc32" (UID: "d885f72a-2f29-41ee-b803-ab1a3a96fc32"). InnerVolumeSpecName "kube-api-access-jv7pc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:37:30.211231 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.211195    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d885f72a-2f29-41ee-b803-ab1a3a96fc32-istio-envoy" pod "d885f72a-2f29-41ee-b803-ab1a3a96fc32" (UID: "d885f72a-2f29-41ee-b803-ab1a3a96fc32"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:37:30.211782 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.211707    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws"
Nov 01 05:37:30.213012 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.212982    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d885f72a-2f29-41ee-b803-ab1a3a96fc32-istio-token" pod "d885f72a-2f29-41ee-b803-ab1a3a96fc32" (UID: "d885f72a-2f29-41ee-b803-ab1a3a96fc32"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:37:30.213665 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.213630    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d885f72a-2f29-41ee-b803-ab1a3a96fc32-istio-podinfo" pod "d885f72a-2f29-41ee-b803-ab1a3a96fc32" (UID: "d885f72a-2f29-41ee-b803-ab1a3a96fc32"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 05:37:30.292680 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.292479    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c7f07f0-82ce-4e5b-97c5-946cc901566d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c7f07f0-82ce-4e5b-97c5-946cc901566d-istio-data\") pod \"4c7f07f0-82ce-4e5b-97c5-946cc901566d\" (UID: \"4c7f07f0-82ce-4e5b-97c5-946cc901566d\") "
Nov 01 05:37:30.292680 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.292537    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0eccaa64-f564-44d7-a611-9ff326b899b6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eccaa64-f564-44d7-a611-9ff326b899b6-workload-socket\") pod \"0eccaa64-f564-44d7-a611-9ff326b899b6\" (UID: \"0eccaa64-f564-44d7-a611-9ff326b899b6\") "
Nov 01 05:37:30.292680 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.292582    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0eccaa64-f564-44d7-a611-9ff326b899b6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0eccaa64-f564-44d7-a611-9ff326b899b6-istio-envoy\") pod \"0eccaa64-f564-44d7-a611-9ff326b899b6\" (UID: \"0eccaa64-f564-44d7-a611-9ff326b899b6\") "
Nov 01 05:37:30.292680 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.292610    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c7f07f0-82ce-4e5b-97c5-946cc901566d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c7f07f0-82ce-4e5b-97c5-946cc901566d-istio-envoy\") pod \"4c7f07f0-82ce-4e5b-97c5-946cc901566d\" (UID: \"4c7f07f0-82ce-4e5b-97c5-946cc901566d\") "
Nov 01 05:37:30.292680 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.292650    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4c7f07f0-82ce-4e5b-97c5-946cc901566d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c7f07f0-82ce-4e5b-97c5-946cc901566d-istio-podinfo\") pod \"4c7f07f0-82ce-4e5b-97c5-946cc901566d\" (UID: \"4c7f07f0-82ce-4e5b-97c5-946cc901566d\") "
Nov 01 05:37:30.292680 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.292688    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0eccaa64-f564-44d7-a611-9ff326b899b6-istio-token\" (UniqueName: \"kubernetes.io/projected/0eccaa64-f564-44d7-a611-9ff326b899b6-istio-token\") pod \"0eccaa64-f564-44d7-a611-9ff326b899b6\" (UID: \"0eccaa64-f564-44d7-a611-9ff326b899b6\") "
Nov 01 05:37:30.293447 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.292733    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c7f07f0-82ce-4e5b-97c5-946cc901566d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c7f07f0-82ce-4e5b-97c5-946cc901566d-workload-certs\") pod \"4c7f07f0-82ce-4e5b-97c5-946cc901566d\" (UID: \"4c7f07f0-82ce-4e5b-97c5-946cc901566d\") "
Nov 01 05:37:30.293447 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.292773    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0eccaa64-f564-44d7-a611-9ff326b899b6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eccaa64-f564-44d7-a611-9ff326b899b6-credential-socket\") pod \"0eccaa64-f564-44d7-a611-9ff326b899b6\" (UID: \"0eccaa64-f564-44d7-a611-9ff326b899b6\") "
Nov 01 05:37:30.293447 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.292801    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0eccaa64-f564-44d7-a611-9ff326b899b6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0eccaa64-f564-44d7-a611-9ff326b899b6-workload-certs\") pod \"0eccaa64-f564-44d7-a611-9ff326b899b6\" (UID: \"0eccaa64-f564-44d7-a611-9ff326b899b6\") "
Nov 01 05:37:30.293447 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.292877    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c7f07f0-82ce-4e5b-97c5-946cc901566d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c7f07f0-82ce-4e5b-97c5-946cc901566d-workload-socket\") pod \"4c7f07f0-82ce-4e5b-97c5-946cc901566d\" (UID: \"4c7f07f0-82ce-4e5b-97c5-946cc901566d\") "
Nov 01 05:37:30.293447 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.292941    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c7f07f0-82ce-4e5b-97c5-946cc901566d-istio-token\" (UniqueName: \"kubernetes.io/projected/4c7f07f0-82ce-4e5b-97c5-946cc901566d-istio-token\") pod \"4c7f07f0-82ce-4e5b-97c5-946cc901566d\" (UID: \"4c7f07f0-82ce-4e5b-97c5-946cc901566d\") "
Nov 01 05:37:30.293447 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.292997    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c7f07f0-82ce-4e5b-97c5-946cc901566d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c7f07f0-82ce-4e5b-97c5-946cc901566d-credential-socket\") pod \"4c7f07f0-82ce-4e5b-97c5-946cc901566d\" (UID: \"4c7f07f0-82ce-4e5b-97c5-946cc901566d\") "
Nov 01 05:37:30.293447 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.293054    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0eccaa64-f564-44d7-a611-9ff326b899b6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0eccaa64-f564-44d7-a611-9ff326b899b6-istiod-ca-cert\") pod \"0eccaa64-f564-44d7-a611-9ff326b899b6\" (UID: \"0eccaa64-f564-44d7-a611-9ff326b899b6\") "
Nov 01 05:37:30.293447 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.293118    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c7f07f0-82ce-4e5b-97c5-946cc901566d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c7f07f0-82ce-4e5b-97c5-946cc901566d-istiod-ca-cert\") pod \"4c7f07f0-82ce-4e5b-97c5-946cc901566d\" (UID: \"4c7f07f0-82ce-4e5b-97c5-946cc901566d\") "
Nov 01 05:37:30.293447 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.293164    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0eccaa64-f564-44d7-a611-9ff326b899b6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0eccaa64-f564-44d7-a611-9ff326b899b6-istio-data\") pod \"0eccaa64-f564-44d7-a611-9ff326b899b6\" (UID: \"0eccaa64-f564-44d7-a611-9ff326b899b6\") "
Nov 01 05:37:30.293447 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.293230    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0eccaa64-f564-44d7-a611-9ff326b899b6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0eccaa64-f564-44d7-a611-9ff326b899b6-istio-podinfo\") pod \"0eccaa64-f564-44d7-a611-9ff326b899b6\" (UID: \"0eccaa64-f564-44d7-a611-9ff326b899b6\") "
Nov 01 05:37:30.293447 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.293265    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c7f07f0-82ce-4e5b-97c5-946cc901566d-kube-api-access-8wpdk\" (UniqueName: \"kubernetes.io/projected/4c7f07f0-82ce-4e5b-97c5-946cc901566d-kube-api-access-8wpdk\") pod \"4c7f07f0-82ce-4e5b-97c5-946cc901566d\" (UID: \"4c7f07f0-82ce-4e5b-97c5-946cc901566d\") "
Nov 01 05:37:30.293447 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.293366    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0eccaa64-f564-44d7-a611-9ff326b899b6-kube-api-access-cc9k6\" (UniqueName: \"kubernetes.io/projected/0eccaa64-f564-44d7-a611-9ff326b899b6-kube-api-access-cc9k6\") pod \"0eccaa64-f564-44d7-a611-9ff326b899b6\" (UID: \"0eccaa64-f564-44d7-a611-9ff326b899b6\") "
Nov 01 05:37:30.294222 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.293745    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eccaa64-f564-44d7-a611-9ff326b899b6-workload-certs" pod "0eccaa64-f564-44d7-a611-9ff326b899b6" (UID: "0eccaa64-f564-44d7-a611-9ff326b899b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:37:30.294222 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.294011    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0eccaa64-f564-44d7-a611-9ff326b899b6-istiod-ca-cert" pod "0eccaa64-f564-44d7-a611-9ff326b899b6" (UID: "0eccaa64-f564-44d7-a611-9ff326b899b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:37:30.294339 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.294292    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c7f07f0-82ce-4e5b-97c5-946cc901566d-workload-socket" pod "4c7f07f0-82ce-4e5b-97c5-946cc901566d" (UID: "4c7f07f0-82ce-4e5b-97c5-946cc901566d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:37:30.294339 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.294309    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c7f07f0-82ce-4e5b-97c5-946cc901566d-istio-data" pod "4c7f07f0-82ce-4e5b-97c5-946cc901566d" (UID: "4c7f07f0-82ce-4e5b-97c5-946cc901566d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:37:30.295856 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.294533    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eccaa64-f564-44d7-a611-9ff326b899b6-workload-socket" pod "0eccaa64-f564-44d7-a611-9ff326b899b6" (UID: "0eccaa64-f564-44d7-a611-9ff326b899b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:37:30.295856 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.294974    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c7f07f0-82ce-4e5b-97c5-946cc901566d-istiod-ca-cert" pod "4c7f07f0-82ce-4e5b-97c5-946cc901566d" (UID: "4c7f07f0-82ce-4e5b-97c5-946cc901566d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:37:30.295856 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.295278    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eccaa64-f564-44d7-a611-9ff326b899b6-istio-data" pod "0eccaa64-f564-44d7-a611-9ff326b899b6" (UID: "0eccaa64-f564-44d7-a611-9ff326b899b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:37:30.295856 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.295597    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c7f07f0-82ce-4e5b-97c5-946cc901566d-workload-certs" pod "4c7f07f0-82ce-4e5b-97c5-946cc901566d" (UID: "4c7f07f0-82ce-4e5b-97c5-946cc901566d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:37:30.296152 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.296117    5689 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d885f72a-2f29-41ee-b803-ab1a3a96fc32-istio-token\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:37:30.296152 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.296139    5689 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c7f07f0-82ce-4e5b-97c5-946cc901566d-istiod-ca-cert\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:37:30.296274 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.296153    5689 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0eccaa64-f564-44d7-a611-9ff326b899b6-istio-data\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:37:30.296274 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.296168    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jv7pc\" (UniqueName: \"kubernetes.io/projected/d885f72a-2f29-41ee-b803-ab1a3a96fc32-kube-api-access-jv7pc\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:37:30.296274 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.296181    5689 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d885f72a-2f29-41ee-b803-ab1a3a96fc32-workload-certs\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:37:30.296274 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.296194    5689 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c7f07f0-82ce-4e5b-97c5-946cc901566d-istio-data\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:37:30.296274 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.296207    5689 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eccaa64-f564-44d7-a611-9ff326b899b6-workload-socket\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:37:30.296274 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.296220    5689 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c7f07f0-82ce-4e5b-97c5-946cc901566d-workload-certs\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:37:30.296274 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.296233    5689 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d885f72a-2f29-41ee-b803-ab1a3a96fc32-istio-data\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:37:30.296274 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.296248    5689 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0eccaa64-f564-44d7-a611-9ff326b899b6-workload-certs\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:37:30.296274 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.296261    5689 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d885f72a-2f29-41ee-b803-ab1a3a96fc32-credential-socket\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:37:30.296274 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.296273    5689 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c7f07f0-82ce-4e5b-97c5-946cc901566d-workload-socket\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:37:30.296839 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.296287    5689 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d885f72a-2f29-41ee-b803-ab1a3a96fc32-istio-envoy\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:37:30.296839 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.296301    5689 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d885f72a-2f29-41ee-b803-ab1a3a96fc32-istio-podinfo\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:37:30.296839 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.296340    5689 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d885f72a-2f29-41ee-b803-ab1a3a96fc32-workload-socket\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:37:30.296839 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.296354    5689 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0eccaa64-f564-44d7-a611-9ff326b899b6-istiod-ca-cert\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:37:30.297065 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.297037    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c7f07f0-82ce-4e5b-97c5-946cc901566d-istio-token" pod "4c7f07f0-82ce-4e5b-97c5-946cc901566d" (UID: "4c7f07f0-82ce-4e5b-97c5-946cc901566d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:37:30.297479 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.297409    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eccaa64-f564-44d7-a611-9ff326b899b6-credential-socket" pod "0eccaa64-f564-44d7-a611-9ff326b899b6" (UID: "0eccaa64-f564-44d7-a611-9ff326b899b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:37:30.298045 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.298023    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c7f07f0-82ce-4e5b-97c5-946cc901566d-credential-socket" pod "4c7f07f0-82ce-4e5b-97c5-946cc901566d" (UID: "4c7f07f0-82ce-4e5b-97c5-946cc901566d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:37:30.298904 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.298871    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c7f07f0-82ce-4e5b-97c5-946cc901566d-kube-api-access-8wpdk" pod "4c7f07f0-82ce-4e5b-97c5-946cc901566d" (UID: "4c7f07f0-82ce-4e5b-97c5-946cc901566d"). InnerVolumeSpecName "kube-api-access-8wpdk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:37:30.299223 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.299197    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eccaa64-f564-44d7-a611-9ff326b899b6-kube-api-access-cc9k6" pod "0eccaa64-f564-44d7-a611-9ff326b899b6" (UID: "0eccaa64-f564-44d7-a611-9ff326b899b6"). InnerVolumeSpecName "kube-api-access-cc9k6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:37:30.300917 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.300882    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4c7f07f0-82ce-4e5b-97c5-946cc901566d-istio-podinfo" pod "4c7f07f0-82ce-4e5b-97c5-946cc901566d" (UID: "4c7f07f0-82ce-4e5b-97c5-946cc901566d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 05:37:30.301245 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.301196    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0eccaa64-f564-44d7-a611-9ff326b899b6-istio-podinfo" pod "0eccaa64-f564-44d7-a611-9ff326b899b6" (UID: "0eccaa64-f564-44d7-a611-9ff326b899b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 05:37:30.301386 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.301294    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eccaa64-f564-44d7-a611-9ff326b899b6-istio-envoy" pod "0eccaa64-f564-44d7-a611-9ff326b899b6" (UID: "0eccaa64-f564-44d7-a611-9ff326b899b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:37:30.301729 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.301704    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c7f07f0-82ce-4e5b-97c5-946cc901566d-istio-envoy" pod "4c7f07f0-82ce-4e5b-97c5-946cc901566d" (UID: "4c7f07f0-82ce-4e5b-97c5-946cc901566d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:37:30.303151 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.303091    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eccaa64-f564-44d7-a611-9ff326b899b6-istio-token" pod "0eccaa64-f564-44d7-a611-9ff326b899b6" (UID: "0eccaa64-f564-44d7-a611-9ff326b899b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:37:30.401436 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.401106    5689 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eccaa64-f564-44d7-a611-9ff326b899b6-credential-socket\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:37:30.401436 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.401146    5689 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c7f07f0-82ce-4e5b-97c5-946cc901566d-istio-token\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:37:30.401436 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.401161    5689 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c7f07f0-82ce-4e5b-97c5-946cc901566d-credential-socket\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:37:30.401436 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.401178    5689 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0eccaa64-f564-44d7-a611-9ff326b899b6-istio-podinfo\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:37:30.401436 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.401193    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wpdk\" (UniqueName: \"kubernetes.io/projected/4c7f07f0-82ce-4e5b-97c5-946cc901566d-kube-api-access-8wpdk\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:37:30.401436 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.401209    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cc9k6\" (UniqueName: \"kubernetes.io/projected/0eccaa64-f564-44d7-a611-9ff326b899b6-kube-api-access-cc9k6\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:37:30.401436 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.401225    5689 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0eccaa64-f564-44d7-a611-9ff326b899b6-istio-envoy\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:37:30.401436 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.401241    5689 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c7f07f0-82ce-4e5b-97c5-946cc901566d-istio-envoy\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:37:30.401436 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.401255    5689 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c7f07f0-82ce-4e5b-97c5-946cc901566d-istio-podinfo\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:37:30.401436 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.401268    5689 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0eccaa64-f564-44d7-a611-9ff326b899b6-istio-token\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:37:30.970517 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.970480    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws" event={"ID":"0eccaa64-f564-44d7-a611-9ff326b899b6","Type":"ContainerDied","Data":"4951cba639a60649727b4c79e78745eba1bf5fb4925a672936bdf779ffffb378"}
Nov 01 05:37:30.970517 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.970503    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws"
Nov 01 05:37:30.971057 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.970581    5689 scope.go:122] "RemoveContainer" containerID="ae218ab3dbdf626581a3f783a88ea00cef604ec42e1fa82ccea0faa4ff0b019d"
Nov 01 05:37:30.972042 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.972022    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl"
Nov 01 05:37:30.972247 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.972208    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl" event={"ID":"4c7f07f0-82ce-4e5b-97c5-946cc901566d","Type":"ContainerDied","Data":"fd39bb03e5df69ec8c727258ca5b8e1ac560f546081a2cb2b483c66ede5b30c9"}
Nov 01 05:37:30.973642 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.973621    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb"
Nov 01 05:37:30.973755 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.973646    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb" event={"ID":"c8a582fb-747c-4c55-a0e3-b5f5139fa28f","Type":"ContainerDied","Data":"3ae51e7ad3332067af9dec10f8a59e3d4df834f5be5545294a2245513c0caee2"}
Nov 01 05:37:30.986475 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.975117    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4" event={"ID":"d885f72a-2f29-41ee-b803-ab1a3a96fc32","Type":"ContainerDied","Data":"5381bed14b6c7e57579e331ccaa76690f38504f1f4343e2828bcba5273f30b46"}
Nov 01 05:37:30.986475 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:30.975223    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4"
Nov 01 05:37:31.051619 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:31.050288    5689 scope.go:122] "RemoveContainer" containerID="a6e7821944ee9614d85fc9b232cf0f63caa1e755014ab9f1eb2029146e0d9415"
Nov 01 05:37:31.071735 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:31.071708    5689 scope.go:122] "RemoveContainer" containerID="977315bedf667832fb45730d9da5f32d29e00bd9ffcc867e2b1fef2900eb0767"
Nov 01 05:37:31.121653 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:31.121340    5689 scope.go:122] "RemoveContainer" containerID="daec85060065e22ccd8f7539888b7f6ec2d75b876fa53258eb3164ff68a3fb6e"
Nov 01 05:37:31.159040 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:37:31.159011    5689 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8a582fb_747c_4c55_a0e3_b5f5139fa28f.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8a582fb_747c_4c55_a0e3_b5f5139fa28f.slice/crio-3ae51e7ad3332067af9dec10f8a59e3d4df834f5be5545294a2245513c0caee2\": RecentStats: unable to find data in memory cache]"
Nov 01 05:37:31.394529 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:31.378069    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb"]
Nov 01 05:37:31.394529 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:31.381833    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-2tbkb"]
Nov 01 05:37:31.502478 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:31.502164    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws"]
Nov 01 05:37:31.507473 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:31.504883    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-pl9ws"]
Nov 01 05:37:31.621480 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:31.620514    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl"]
Nov 01 05:37:31.624625 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:31.624424    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-l62wl"]
Nov 01 05:37:31.738465 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:31.738412    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4"]
Nov 01 05:37:31.743230 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:31.743205    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-wdzh4"]
Nov 01 05:37:32.056813 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:32.056710    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:37:32.084481 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:32.071501    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0eccaa64-f564-44d7-a611-9ff326b899b6" path="/var/lib/kubelet/pods/0eccaa64-f564-44d7-a611-9ff326b899b6/volumes"
Nov 01 05:37:32.084481 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:32.072211    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c7f07f0-82ce-4e5b-97c5-946cc901566d" path="/var/lib/kubelet/pods/4c7f07f0-82ce-4e5b-97c5-946cc901566d/volumes"
Nov 01 05:37:32.084481 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:32.073054    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8a582fb-747c-4c55-a0e3-b5f5139fa28f" path="/var/lib/kubelet/pods/c8a582fb-747c-4c55-a0e3-b5f5139fa28f/volumes"
Nov 01 05:37:32.099478 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:32.091207    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d885f72a-2f29-41ee-b803-ab1a3a96fc32" path="/var/lib/kubelet/pods/d885f72a-2f29-41ee-b803-ab1a3a96fc32/volumes"
Nov 01 05:37:34.892781 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:34.892719    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:37:35.114355 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:35.114319    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:37:36.851028 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:36.850977    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:37:41.598944 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:41.598902    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:37:44.326208 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:44.326175    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:37:46.519490 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:46.519437    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:37:51.315810 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:51.315776    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:37:56.097566 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:37:56.097534    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:38:00.068743 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:38:00.068707    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:38:00.285371 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:00.285338    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:38:01.780527 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:01.780488    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:38:02.000608 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:02.000578    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:38:02.174204 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:02.174175    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:38:02.599783 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:02.599715    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:38:03.159895 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:03.159868    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:38:03.640542 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:03.640433    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664"]
Nov 01 05:38:03.641071 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:03.641052    5689 cpu_manager.go:414] "containerMap: removing container" podUID="2a89bfe1-3a8a-4fd6-9b3b-d02e3c0695dc" containerName="payload-processing"
Nov 01 05:38:03.641176 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:03.641074    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a89bfe1-3a8a-4fd6-9b3b-d02e3c0695dc" containerName="payload-processing"
Nov 01 05:38:03.641176 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:03.641097    5689 cpu_manager.go:414] "containerMap: removing container" podUID="c8a582fb-747c-4c55-a0e3-b5f5139fa28f" containerName="istio-proxy"
Nov 01 05:38:03.641176 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:03.641104    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8a582fb-747c-4c55-a0e3-b5f5139fa28f" containerName="istio-proxy"
Nov 01 05:38:03.641176 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:03.641128    5689 cpu_manager.go:414] "containerMap: removing container" podUID="4c7f07f0-82ce-4e5b-97c5-946cc901566d" containerName="istio-proxy"
Nov 01 05:38:03.641176 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:03.641136    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c7f07f0-82ce-4e5b-97c5-946cc901566d" containerName="istio-proxy"
Nov 01 05:38:03.641176 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:03.641157    5689 cpu_manager.go:414] "containerMap: removing container" podUID="c8a582fb-747c-4c55-a0e3-b5f5139fa28f" containerName="istio-proxy"
Nov 01 05:38:03.641176 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:03.641164    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8a582fb-747c-4c55-a0e3-b5f5139fa28f" containerName="istio-proxy"
Nov 01 05:38:03.641565 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:03.641186    5689 cpu_manager.go:414] "containerMap: removing container" podUID="d885f72a-2f29-41ee-b803-ab1a3a96fc32" containerName="istio-proxy"
Nov 01 05:38:03.641565 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:03.641194    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d885f72a-2f29-41ee-b803-ab1a3a96fc32" containerName="istio-proxy"
Nov 01 05:38:03.641565 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:03.641208    5689 cpu_manager.go:414] "containerMap: removing container" podUID="0eccaa64-f564-44d7-a611-9ff326b899b6" containerName="istio-proxy"
Nov 01 05:38:03.641565 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:03.641215    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0eccaa64-f564-44d7-a611-9ff326b899b6" containerName="istio-proxy"
Nov 01 05:38:03.641565 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:03.641372    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a89bfe1-3a8a-4fd6-9b3b-d02e3c0695dc" containerName="payload-processing"
Nov 01 05:38:03.641565 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:03.641389    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8a582fb-747c-4c55-a0e3-b5f5139fa28f" containerName="istio-proxy"
Nov 01 05:38:03.641565 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:03.641400    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c7f07f0-82ce-4e5b-97c5-946cc901566d" containerName="istio-proxy"
Nov 01 05:38:03.641565 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:03.641414    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8a582fb-747c-4c55-a0e3-b5f5139fa28f" containerName="istio-proxy"
Nov 01 05:38:03.641565 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:03.641425    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="d885f72a-2f29-41ee-b803-ab1a3a96fc32" containerName="istio-proxy"
Nov 01 05:38:03.641565 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:03.641433    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="0eccaa64-f564-44d7-a611-9ff326b899b6" containerName="istio-proxy"
Nov 01 05:38:03.644835 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:03.644809    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664"
Nov 01 05:38:03.708475 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:03.699356    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-4cbcb62b-openshift-default-dockercfg-5x9gj\""
Nov 01 05:38:03.755414 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:03.751531    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a009de5-76be-4db1-adec-efb9246d4ddf-credential-socket\") pod \"e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664\" (UID: \"9a009de5-76be-4db1-adec-efb9246d4ddf\") " pod="openshift-ingress/e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664"
Nov 01 05:38:03.755414 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:03.752983    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664"]
Nov 01 05:38:03.755414 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:03.751627    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a009de5-76be-4db1-adec-efb9246d4ddf-istio-podinfo\") pod \"e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664\" (UID: \"9a009de5-76be-4db1-adec-efb9246d4ddf\") " pod="openshift-ingress/e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664"
Nov 01 05:38:03.755414 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:03.754605    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pc8lp\" (UniqueName: \"kubernetes.io/projected/9a009de5-76be-4db1-adec-efb9246d4ddf-kube-api-access-pc8lp\") pod \"e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664\" (UID: \"9a009de5-76be-4db1-adec-efb9246d4ddf\") " pod="openshift-ingress/e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664"
Nov 01 05:38:03.755414 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:03.754656    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a009de5-76be-4db1-adec-efb9246d4ddf-workload-certs\") pod \"e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664\" (UID: \"9a009de5-76be-4db1-adec-efb9246d4ddf\") " pod="openshift-ingress/e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664"
Nov 01 05:38:03.755414 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:03.754702    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a009de5-76be-4db1-adec-efb9246d4ddf-istio-data\") pod \"e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664\" (UID: \"9a009de5-76be-4db1-adec-efb9246d4ddf\") " pod="openshift-ingress/e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664"
Nov 01 05:38:03.755414 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:03.754733    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a009de5-76be-4db1-adec-efb9246d4ddf-istiod-ca-cert\") pod \"e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664\" (UID: \"9a009de5-76be-4db1-adec-efb9246d4ddf\") " pod="openshift-ingress/e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664"
Nov 01 05:38:03.755414 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:03.754786    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a009de5-76be-4db1-adec-efb9246d4ddf-workload-socket\") pod \"e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664\" (UID: \"9a009de5-76be-4db1-adec-efb9246d4ddf\") " pod="openshift-ingress/e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664"
Nov 01 05:38:03.755414 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:03.754819    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a009de5-76be-4db1-adec-efb9246d4ddf-istio-token\") pod \"e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664\" (UID: \"9a009de5-76be-4db1-adec-efb9246d4ddf\") " pod="openshift-ingress/e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664"
Nov 01 05:38:03.755414 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:03.754907    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a009de5-76be-4db1-adec-efb9246d4ddf-istio-envoy\") pod \"e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664\" (UID: \"9a009de5-76be-4db1-adec-efb9246d4ddf\") " pod="openshift-ingress/e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664"
Nov 01 05:38:03.772468 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:03.772381    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:38:03.857886 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:03.857160    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a009de5-76be-4db1-adec-efb9246d4ddf-istio-data\") pod \"e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664\" (UID: \"9a009de5-76be-4db1-adec-efb9246d4ddf\") " pod="openshift-ingress/e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664"
Nov 01 05:38:03.857886 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:03.857200    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a009de5-76be-4db1-adec-efb9246d4ddf-istiod-ca-cert\") pod \"e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664\" (UID: \"9a009de5-76be-4db1-adec-efb9246d4ddf\") " pod="openshift-ingress/e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664"
Nov 01 05:38:03.857886 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:03.857235    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a009de5-76be-4db1-adec-efb9246d4ddf-workload-socket\") pod \"e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664\" (UID: \"9a009de5-76be-4db1-adec-efb9246d4ddf\") " pod="openshift-ingress/e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664"
Nov 01 05:38:03.857886 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:03.857256    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a009de5-76be-4db1-adec-efb9246d4ddf-istio-token\") pod \"e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664\" (UID: \"9a009de5-76be-4db1-adec-efb9246d4ddf\") " pod="openshift-ingress/e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664"
Nov 01 05:38:03.857886 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:03.857306    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a009de5-76be-4db1-adec-efb9246d4ddf-istio-envoy\") pod \"e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664\" (UID: \"9a009de5-76be-4db1-adec-efb9246d4ddf\") " pod="openshift-ingress/e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664"
Nov 01 05:38:03.857886 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:03.857362    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a009de5-76be-4db1-adec-efb9246d4ddf-credential-socket\") pod \"e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664\" (UID: \"9a009de5-76be-4db1-adec-efb9246d4ddf\") " pod="openshift-ingress/e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664"
Nov 01 05:38:03.857886 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:03.857435    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a009de5-76be-4db1-adec-efb9246d4ddf-istio-podinfo\") pod \"e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664\" (UID: \"9a009de5-76be-4db1-adec-efb9246d4ddf\") " pod="openshift-ingress/e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664"
Nov 01 05:38:03.857886 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:03.857472    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc8lp\" (UniqueName: \"kubernetes.io/projected/9a009de5-76be-4db1-adec-efb9246d4ddf-kube-api-access-pc8lp\") pod \"e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664\" (UID: \"9a009de5-76be-4db1-adec-efb9246d4ddf\") " pod="openshift-ingress/e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664"
Nov 01 05:38:03.857886 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:03.857506    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a009de5-76be-4db1-adec-efb9246d4ddf-workload-certs\") pod \"e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664\" (UID: \"9a009de5-76be-4db1-adec-efb9246d4ddf\") " pod="openshift-ingress/e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664"
Nov 01 05:38:03.857886 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:03.857818    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a009de5-76be-4db1-adec-efb9246d4ddf-workload-certs\") pod \"e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664\" (UID: \"9a009de5-76be-4db1-adec-efb9246d4ddf\") " pod="openshift-ingress/e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664"
Nov 01 05:38:03.860763 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:03.860725    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a009de5-76be-4db1-adec-efb9246d4ddf-credential-socket\") pod \"e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664\" (UID: \"9a009de5-76be-4db1-adec-efb9246d4ddf\") " pod="openshift-ingress/e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664"
Nov 01 05:38:03.860874 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:03.860844    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a009de5-76be-4db1-adec-efb9246d4ddf-istiod-ca-cert\") pod \"e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664\" (UID: \"9a009de5-76be-4db1-adec-efb9246d4ddf\") " pod="openshift-ingress/e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664"
Nov 01 05:38:03.862012 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:03.861991    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a009de5-76be-4db1-adec-efb9246d4ddf-workload-socket\") pod \"e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664\" (UID: \"9a009de5-76be-4db1-adec-efb9246d4ddf\") " pod="openshift-ingress/e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664"
Nov 01 05:38:03.862336 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:03.862198    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a009de5-76be-4db1-adec-efb9246d4ddf-istio-envoy\") pod \"e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664\" (UID: \"9a009de5-76be-4db1-adec-efb9246d4ddf\") " pod="openshift-ingress/e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664"
Nov 01 05:38:03.862336 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:03.862265    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a009de5-76be-4db1-adec-efb9246d4ddf-istio-data\") pod \"e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664\" (UID: \"9a009de5-76be-4db1-adec-efb9246d4ddf\") " pod="openshift-ingress/e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664"
Nov 01 05:38:03.864248 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:03.864219    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a009de5-76be-4db1-adec-efb9246d4ddf-istio-podinfo\") pod \"e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664\" (UID: \"9a009de5-76be-4db1-adec-efb9246d4ddf\") " pod="openshift-ingress/e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664"
Nov 01 05:38:03.918599 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:03.918575    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a009de5-76be-4db1-adec-efb9246d4ddf-istio-token\") pod \"e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664\" (UID: \"9a009de5-76be-4db1-adec-efb9246d4ddf\") " pod="openshift-ingress/e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664"
Nov 01 05:38:03.918787 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:03.918766    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc8lp\" (UniqueName: \"kubernetes.io/projected/9a009de5-76be-4db1-adec-efb9246d4ddf-kube-api-access-pc8lp\") pod \"e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664\" (UID: \"9a009de5-76be-4db1-adec-efb9246d4ddf\") " pod="openshift-ingress/e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664"
Nov 01 05:38:03.957703 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:03.957681    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664"
Nov 01 05:38:04.034119 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:04.033925    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:38:04.411375 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:04.411345    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664"]
Nov 01 05:38:04.414668 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:38:04.414640    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a009de5_76be_4db1_adec_efb9246d4ddf.slice/crio-5ad2a62497190b33b1e20c2db3720de321402ccfcb47a20e8947343ee98993a6 WatchSource:0}: Error finding container 5ad2a62497190b33b1e20c2db3720de321402ccfcb47a20e8947343ee98993a6: Status 404 returned error can't find the container with id 5ad2a62497190b33b1e20c2db3720de321402ccfcb47a20e8947343ee98993a6
Nov 01 05:38:04.416945 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:04.416915    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:38:04.417014 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:04.416978    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:38:04.417014 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:04.417007    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:38:05.188788 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:05.188750    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664" event={"ID":"9a009de5-76be-4db1-adec-efb9246d4ddf","Type":"ContainerStarted","Data":"fce93fd45f1a7f3f0bbb753a9fee3c2f9ffaab23e93fe6e9021ea2c411352c9c"}
Nov 01 05:38:05.188999 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:05.188799    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664" event={"ID":"9a009de5-76be-4db1-adec-efb9246d4ddf","Type":"ContainerStarted","Data":"5ad2a62497190b33b1e20c2db3720de321402ccfcb47a20e8947343ee98993a6"}
Nov 01 05:38:05.326527 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:05.326493    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:38:05.355480 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:05.355422    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664" podStartSLOduration=2.35540791 podStartE2EDuration="2.35540791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:38:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:38:05.329090422 +0000 UTC m=+2063.399887141" watchObservedRunningTime="2026-11-01 05:38:05.35540791 +0000 UTC m=+2063.426204629"
Nov 01 05:38:05.957946 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:05.957915    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664"
Nov 01 05:38:05.963634 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:05.963609    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664"
Nov 01 05:38:06.193975 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:06.193938    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664"
Nov 01 05:38:06.194743 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:06.194723    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664"
Nov 01 05:38:07.413962 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:07.413936    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:38:07.599373 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:07.599335    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:38:10.646073 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:10.646037    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:38:10.828388 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:10.828352    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:38:11.032401 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:11.032367    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4cbcb62b-5d57bdcc59-vpvfg"]
Nov 01 05:38:11.036489 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:11.036445    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4cbcb62b-5d57bdcc59-vpvfg"
Nov 01 05:38:11.089626 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:11.089603    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-4cbcb62b\""
Nov 01 05:38:11.153707 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:11.153671    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-4cbcb62b-dockercfg-7c5t7\""
Nov 01 05:38:11.174481 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:11.161584    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b449f8f-a42d-4645-a555-3c5727026a47-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4cbcb62b-5d57bdcc59-vpvfg\" (UID: \"1b449f8f-a42d-4645-a555-3c5727026a47\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4cbcb62b-5d57bdcc59-vpvfg"
Nov 01 05:38:11.174481 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:11.161785    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lt8m\" (UniqueName: \"kubernetes.io/projected/1b449f8f-a42d-4645-a555-3c5727026a47-kube-api-access-2lt8m\") pod \"payload-pre-processing-e2e-ait-4cbcb62b-5d57bdcc59-vpvfg\" (UID: \"1b449f8f-a42d-4645-a555-3c5727026a47\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4cbcb62b-5d57bdcc59-vpvfg"
Nov 01 05:38:11.174481 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:11.161966    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4cbcb62b-5d57bdcc59-vpvfg"]
Nov 01 05:38:11.185849 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:11.185816    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4cbcb62b-849d7dcbc9-vjtls"]
Nov 01 05:38:11.211834 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:11.211790    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4cbcb62b-849d7dcbc9-vjtls"
Nov 01 05:38:11.243690 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:11.243662    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4cbcb62b-849d7dcbc9-vjtls"]
Nov 01 05:38:11.263005 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:11.262976    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lt8m\" (UniqueName: \"kubernetes.io/projected/1b449f8f-a42d-4645-a555-3c5727026a47-kube-api-access-2lt8m\") pod \"payload-pre-processing-e2e-ait-4cbcb62b-5d57bdcc59-vpvfg\" (UID: \"1b449f8f-a42d-4645-a555-3c5727026a47\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4cbcb62b-5d57bdcc59-vpvfg"
Nov 01 05:38:11.263165 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:11.263077    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b449f8f-a42d-4645-a555-3c5727026a47-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4cbcb62b-5d57bdcc59-vpvfg\" (UID: \"1b449f8f-a42d-4645-a555-3c5727026a47\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4cbcb62b-5d57bdcc59-vpvfg"
Nov 01 05:38:11.263234 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:11.263166    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmwrb\" (UniqueName: \"kubernetes.io/projected/e2f0666e-d262-4206-ae1f-b4feeeb56c0d-kube-api-access-dmwrb\") pod \"payload-processing-e2e-ait-4cbcb62b-849d7dcbc9-vjtls\" (UID: \"e2f0666e-d262-4206-ae1f-b4feeeb56c0d\") " pod="openshift-ingress/payload-processing-e2e-ait-4cbcb62b-849d7dcbc9-vjtls"
Nov 01 05:38:11.263310 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:11.263233    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2f0666e-d262-4206-ae1f-b4feeeb56c0d-plugins-config-volume\") pod \"payload-processing-e2e-ait-4cbcb62b-849d7dcbc9-vjtls\" (UID: \"e2f0666e-d262-4206-ae1f-b4feeeb56c0d\") " pod="openshift-ingress/payload-processing-e2e-ait-4cbcb62b-849d7dcbc9-vjtls"
Nov 01 05:38:11.263785 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:11.263767    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b449f8f-a42d-4645-a555-3c5727026a47-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4cbcb62b-5d57bdcc59-vpvfg\" (UID: \"1b449f8f-a42d-4645-a555-3c5727026a47\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4cbcb62b-5d57bdcc59-vpvfg"
Nov 01 05:38:11.324212 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:11.324154    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lt8m\" (UniqueName: \"kubernetes.io/projected/1b449f8f-a42d-4645-a555-3c5727026a47-kube-api-access-2lt8m\") pod \"payload-pre-processing-e2e-ait-4cbcb62b-5d57bdcc59-vpvfg\" (UID: \"1b449f8f-a42d-4645-a555-3c5727026a47\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4cbcb62b-5d57bdcc59-vpvfg"
Nov 01 05:38:11.351297 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:11.351273    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4cbcb62b-5d57bdcc59-vpvfg"
Nov 01 05:38:11.364551 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:11.364522    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmwrb\" (UniqueName: \"kubernetes.io/projected/e2f0666e-d262-4206-ae1f-b4feeeb56c0d-kube-api-access-dmwrb\") pod \"payload-processing-e2e-ait-4cbcb62b-849d7dcbc9-vjtls\" (UID: \"e2f0666e-d262-4206-ae1f-b4feeeb56c0d\") " pod="openshift-ingress/payload-processing-e2e-ait-4cbcb62b-849d7dcbc9-vjtls"
Nov 01 05:38:11.364655 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:11.364605    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2f0666e-d262-4206-ae1f-b4feeeb56c0d-plugins-config-volume\") pod \"payload-processing-e2e-ait-4cbcb62b-849d7dcbc9-vjtls\" (UID: \"e2f0666e-d262-4206-ae1f-b4feeeb56c0d\") " pod="openshift-ingress/payload-processing-e2e-ait-4cbcb62b-849d7dcbc9-vjtls"
Nov 01 05:38:11.365425 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:11.365404    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2f0666e-d262-4206-ae1f-b4feeeb56c0d-plugins-config-volume\") pod \"payload-processing-e2e-ait-4cbcb62b-849d7dcbc9-vjtls\" (UID: \"e2f0666e-d262-4206-ae1f-b4feeeb56c0d\") " pod="openshift-ingress/payload-processing-e2e-ait-4cbcb62b-849d7dcbc9-vjtls"
Nov 01 05:38:11.432288 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:11.432261    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmwrb\" (UniqueName: \"kubernetes.io/projected/e2f0666e-d262-4206-ae1f-b4feeeb56c0d-kube-api-access-dmwrb\") pod \"payload-processing-e2e-ait-4cbcb62b-849d7dcbc9-vjtls\" (UID: \"e2f0666e-d262-4206-ae1f-b4feeeb56c0d\") " pod="openshift-ingress/payload-processing-e2e-ait-4cbcb62b-849d7dcbc9-vjtls"
Nov 01 05:38:11.535139 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:11.535112    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4cbcb62b-849d7dcbc9-vjtls"
Nov 01 05:38:11.725817 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:11.725785    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4cbcb62b-5d57bdcc59-vpvfg"]
Nov 01 05:38:11.729231 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:38:11.729197    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b449f8f_a42d_4645_a555_3c5727026a47.slice/crio-4710fd3f6304abdda499df2c5faf080055c9795b1a5f95a02bc29c5eefb437e0 WatchSource:0}: Error finding container 4710fd3f6304abdda499df2c5faf080055c9795b1a5f95a02bc29c5eefb437e0: Status 404 returned error can't find the container with id 4710fd3f6304abdda499df2c5faf080055c9795b1a5f95a02bc29c5eefb437e0
Nov 01 05:38:11.912079 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:11.912059    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4cbcb62b-849d7dcbc9-vjtls"]
Nov 01 05:38:11.914211 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:38:11.914188    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2f0666e_d262_4206_ae1f_b4feeeb56c0d.slice/crio-01ee885431540a2310ff89f14d9eb5a9ccb4cafc497fb47bdb617bbdbe3f6468 WatchSource:0}: Error finding container 01ee885431540a2310ff89f14d9eb5a9ccb4cafc497fb47bdb617bbdbe3f6468: Status 404 returned error can't find the container with id 01ee885431540a2310ff89f14d9eb5a9ccb4cafc497fb47bdb617bbdbe3f6468
Nov 01 05:38:12.233839 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:12.233757    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4cbcb62b-849d7dcbc9-vjtls" event={"ID":"e2f0666e-d262-4206-ae1f-b4feeeb56c0d","Type":"ContainerStarted","Data":"710a8ef6101b070518de8fb0a3d549a4448f268a75495d290ac4861c81d3afdd"}
Nov 01 05:38:12.233839 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:12.233810    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4cbcb62b-849d7dcbc9-vjtls" event={"ID":"e2f0666e-d262-4206-ae1f-b4feeeb56c0d","Type":"ContainerStarted","Data":"01ee885431540a2310ff89f14d9eb5a9ccb4cafc497fb47bdb617bbdbe3f6468"}
Nov 01 05:38:12.234066 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:12.233903    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-4cbcb62b-849d7dcbc9-vjtls"
Nov 01 05:38:12.235387 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:12.235361    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4cbcb62b-5d57bdcc59-vpvfg" event={"ID":"1b449f8f-a42d-4645-a555-3c5727026a47","Type":"ContainerStarted","Data":"57712c15953579e62469c06c2eabaa0f7727455ab84100ca4efa9ba08f71162c"}
Nov 01 05:38:12.235530 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:12.235408    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-4cbcb62b-5d57bdcc59-vpvfg"
Nov 01 05:38:12.235530 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:12.235428    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4cbcb62b-5d57bdcc59-vpvfg" event={"ID":"1b449f8f-a42d-4645-a555-3c5727026a47","Type":"ContainerStarted","Data":"4710fd3f6304abdda499df2c5faf080055c9795b1a5f95a02bc29c5eefb437e0"}
Nov 01 05:38:12.351638 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:12.351590    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-4cbcb62b-849d7dcbc9-vjtls" podStartSLOduration=1.35157588 podStartE2EDuration="1.35157588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:38:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:38:12.349894091 +0000 UTC m=+2070.420690812" watchObservedRunningTime="2026-11-01 05:38:12.35157588 +0000 UTC m=+2070.422372601"
Nov 01 05:38:12.414289 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:12.414254    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:38:12.491533 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:12.478762    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-4cbcb62b-5d57bdcc59-vpvfg" podStartSLOduration=2.478744278 podStartE2EDuration="2.478744278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:38:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:38:12.478212826 +0000 UTC m=+2070.549009560" watchObservedRunningTime="2026-11-01 05:38:12.478744278 +0000 UTC m=+2070.549541000"
Nov 01 05:38:14.420585 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:14.420552    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:38:17.190795 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:17.190762    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:38:21.910438 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:21.910359    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:38:23.244610 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:23.244577    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-4cbcb62b-5d57bdcc59-vpvfg"
Nov 01 05:38:23.245059 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:23.244777    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-4cbcb62b-849d7dcbc9-vjtls"
Nov 01 05:38:26.716098 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:26.716069    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:38:30.069709 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:38:30.069676    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:38:31.484791 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:31.484762    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:38:36.336559 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:36.336508    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:38:41.114676 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:41.114642    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:38:42.781446 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:42.781408    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:38:43.439492 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:43.436503    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:38:43.778890 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:43.778818    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:38:45.921619 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:45.921588    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:38:47.779038 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:47.779001    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:38:47.957720 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:47.957688    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:38:48.142311 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:48.142227    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:38:51.122880 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:51.122846    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:38:51.309444 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:51.309414    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:38:51.656827 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:51.656793    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a08eb536-768db898c5-4m5l2"]
Nov 01 05:38:51.664660 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:51.664630    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a08eb536-768db898c5-4m5l2"
Nov 01 05:38:51.670239 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:51.669999    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a08eb536-768db898c5-4m5l2"]
Nov 01 05:38:51.742350 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:51.742317    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ll8l\" (UniqueName: \"kubernetes.io/projected/9831d79c-480d-4510-9862-dd03e537e768-kube-api-access-4ll8l\") pod \"payload-processing-e2e-mt-a08eb536-768db898c5-4m5l2\" (UID: \"9831d79c-480d-4510-9862-dd03e537e768\") " pod="openshift-ingress/payload-processing-e2e-mt-a08eb536-768db898c5-4m5l2"
Nov 01 05:38:51.742555 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:51.742380    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9831d79c-480d-4510-9862-dd03e537e768-plugins-config-volume\") pod \"payload-processing-e2e-mt-a08eb536-768db898c5-4m5l2\" (UID: \"9831d79c-480d-4510-9862-dd03e537e768\") " pod="openshift-ingress/payload-processing-e2e-mt-a08eb536-768db898c5-4m5l2"
Nov 01 05:38:51.766078 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:51.766049    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-a08eb536\""
Nov 01 05:38:51.766078 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:51.766049    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-a08eb536-dockercfg-bpthx\""
Nov 01 05:38:51.843682 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:51.843656    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9831d79c-480d-4510-9862-dd03e537e768-plugins-config-volume\") pod \"payload-processing-e2e-mt-a08eb536-768db898c5-4m5l2\" (UID: \"9831d79c-480d-4510-9862-dd03e537e768\") " pod="openshift-ingress/payload-processing-e2e-mt-a08eb536-768db898c5-4m5l2"
Nov 01 05:38:51.843799 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:51.843778    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ll8l\" (UniqueName: \"kubernetes.io/projected/9831d79c-480d-4510-9862-dd03e537e768-kube-api-access-4ll8l\") pod \"payload-processing-e2e-mt-a08eb536-768db898c5-4m5l2\" (UID: \"9831d79c-480d-4510-9862-dd03e537e768\") " pod="openshift-ingress/payload-processing-e2e-mt-a08eb536-768db898c5-4m5l2"
Nov 01 05:38:51.844349 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:51.844331    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9831d79c-480d-4510-9862-dd03e537e768-plugins-config-volume\") pod \"payload-processing-e2e-mt-a08eb536-768db898c5-4m5l2\" (UID: \"9831d79c-480d-4510-9862-dd03e537e768\") " pod="openshift-ingress/payload-processing-e2e-mt-a08eb536-768db898c5-4m5l2"
Nov 01 05:38:51.904707 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:51.904687    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ll8l\" (UniqueName: \"kubernetes.io/projected/9831d79c-480d-4510-9862-dd03e537e768-kube-api-access-4ll8l\") pod \"payload-processing-e2e-mt-a08eb536-768db898c5-4m5l2\" (UID: \"9831d79c-480d-4510-9862-dd03e537e768\") " pod="openshift-ingress/payload-processing-e2e-mt-a08eb536-768db898c5-4m5l2"
Nov 01 05:38:51.981484 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:51.981395    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a08eb536-768db898c5-4m5l2"
Nov 01 05:38:52.354664 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:52.354642    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a08eb536-768db898c5-4m5l2"]
Nov 01 05:38:52.357402 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:38:52.357362    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9831d79c_480d_4510_9862_dd03e537e768.slice/crio-27df4138417f9c21c5ce6b61c0a0f55841ec71419f3025868f735a30601abf53 WatchSource:0}: Error finding container 27df4138417f9c21c5ce6b61c0a0f55841ec71419f3025868f735a30601abf53: Status 404 returned error can't find the container with id 27df4138417f9c21c5ce6b61c0a0f55841ec71419f3025868f735a30601abf53
Nov 01 05:38:52.473206 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:52.473166    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a08eb536-768db898c5-4m5l2" event={"ID":"9831d79c-480d-4510-9862-dd03e537e768","Type":"ContainerStarted","Data":"c59676e67c376b28415a6ad6ffa1394daf42e77587681743d85d7f8d0dfba54d"}
Nov 01 05:38:52.473361 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:52.473230    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a08eb536-768db898c5-4m5l2" event={"ID":"9831d79c-480d-4510-9862-dd03e537e768","Type":"ContainerStarted","Data":"27df4138417f9c21c5ce6b61c0a0f55841ec71419f3025868f735a30601abf53"}
Nov 01 05:38:52.473361 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:52.473347    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-a08eb536-768db898c5-4m5l2"
Nov 01 05:38:52.591609 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:52.591566    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-a08eb536-768db898c5-4m5l2" podStartSLOduration=1.591550925 podStartE2EDuration="1.591550925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:38:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:38:52.588677784 +0000 UTC m=+2110.659474505" watchObservedRunningTime="2026-11-01 05:38:52.591550925 +0000 UTC m=+2110.662347647"
Nov 01 05:38:56.198266 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:56.198226    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:38:56.385831 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:56.385797    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:38:56.566621 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:38:56.566542    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:39:00.070373 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:39:00.070342    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:39:01.216509 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:01.214090    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:39:03.481625 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:03.481549    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-a08eb536-768db898c5-4m5l2"
Nov 01 05:39:05.676389 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:05.676356    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:39:06.013748 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:06.013638    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:39:10.788840 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:10.788807    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:39:15.663714 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:15.663680    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:39:19.437187 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:19.437151    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:39:19.631327 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:19.631292    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:39:19.831526 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:19.827582    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:39:20.506964 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:20.506933    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:39:24.821002 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:24.820923    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:39:25.328862 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:25.328828    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:39:29.683092 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:29.683061    5689 generic.go:356] "Generic (PLEG): container finished" podID="9a009de5-76be-4db1-adec-efb9246d4ddf" containerID="fce93fd45f1a7f3f0bbb753a9fee3c2f9ffaab23e93fe6e9021ea2c411352c9c" exitCode=0
Nov 01 05:39:29.683539 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:29.683137    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664" event={"ID":"9a009de5-76be-4db1-adec-efb9246d4ddf","Type":"ContainerDied","Data":"fce93fd45f1a7f3f0bbb753a9fee3c2f9ffaab23e93fe6e9021ea2c411352c9c"}
Nov 01 05:39:29.683593 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:29.683543    5689 scope.go:122] "RemoveContainer" containerID="fce93fd45f1a7f3f0bbb753a9fee3c2f9ffaab23e93fe6e9021ea2c411352c9c"
Nov 01 05:39:29.684953 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:29.684927    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:39:29.685016 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:29.684993    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:39:29.685052 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:29.685026    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:39:30.069739 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:39:30.069655    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:39:30.116715 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:30.116680    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:39:30.308578 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:30.307609    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:39:30.492843 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:30.492817    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:39:30.691379 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:30.691343    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664" event={"ID":"9a009de5-76be-4db1-adec-efb9246d4ddf","Type":"ContainerStarted","Data":"615b7abd04ba4d081fe29d39a77bbe982cd8a97e4ca68d46e26c856b8c2b94b3"}
Nov 01 05:39:30.958405 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:30.958376    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664"
Nov 01 05:39:30.958613 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:30.958538    5689 patch_prober.go:29] interesting pod/e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 01 05:39:30.958692 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:30.958602    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664" podUID="9a009de5-76be-4db1-adec-efb9246d4ddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 01 05:39:31.410477 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:31.409046    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:39:31.958381 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:31.958343    5689 patch_prober.go:29] interesting pod/e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 01 05:39:31.958818 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:31.958396    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664" podUID="9a009de5-76be-4db1-adec-efb9246d4ddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 01 05:39:32.957965 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:32.957931    5689 patch_prober.go:29] interesting pod/e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 01 05:39:32.958168 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:32.957988    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664" podUID="9a009de5-76be-4db1-adec-efb9246d4ddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 01 05:39:33.958823 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:33.958792    5689 patch_prober.go:29] interesting pod/e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 01 05:39:33.959224 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:33.958855    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664" podUID="9a009de5-76be-4db1-adec-efb9246d4ddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 01 05:39:34.959001 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:34.958967    5689 patch_prober.go:29] interesting pod/e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 01 05:39:34.959540 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:34.959031    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664" podUID="9a009de5-76be-4db1-adec-efb9246d4ddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 01 05:39:35.038330 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:35.038294    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:39:35.958967 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:35.958929    5689 patch_prober.go:29] interesting pod/e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 01 05:39:35.959148 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:35.958992    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664" podUID="9a009de5-76be-4db1-adec-efb9246d4ddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 01 05:39:36.198579 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:36.198542    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664"
Nov 01 05:39:36.963530 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:36.963499    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664"
Nov 01 05:39:36.964602 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:36.964581    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664"
Nov 01 05:39:37.219284 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:37.219193    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:39:37.437870 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:37.437842    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:39:39.815209 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:39.815179    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:39:43.084617 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:43.084560    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:39:43.264222 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:43.264191    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:39:43.651991 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:43.651959    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:39:44.090440 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:44.090358    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:39:44.256075 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:44.256044    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6xsddz"]
Nov 01 05:39:44.260087 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:44.260063    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6xsddz"
Nov 01 05:39:44.312847 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:44.312816    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2fd4c9\"/\"openshift-service-ca.crt\""
Nov 01 05:39:44.312974 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:44.312879    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2fd4c9\"/\"payload-processing-external-model-plugins-e2e-extmod-b-2fd4c9\""
Nov 01 05:39:44.332223 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:44.332191    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/52a16023-0c79-474b-ae68-42eaa5d9d452-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6xsddz\" (UID: \"52a16023-0c79-474b-ae68-42eaa5d9d452\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6xsddz"
Nov 01 05:39:44.332359 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:44.332239    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5lnv\" (UniqueName: \"kubernetes.io/projected/52a16023-0c79-474b-ae68-42eaa5d9d452-kube-api-access-z5lnv\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6xsddz\" (UID: \"52a16023-0c79-474b-ae68-42eaa5d9d452\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6xsddz"
Nov 01 05:39:44.332359 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:44.332262    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52a16023-0c79-474b-ae68-42eaa5d9d452-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6xsddz\" (UID: \"52a16023-0c79-474b-ae68-42eaa5d9d452\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6xsddz"
Nov 01 05:39:44.332359 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:44.332305    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/52a16023-0c79-474b-ae68-42eaa5d9d452-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6xsddz\" (UID: \"52a16023-0c79-474b-ae68-42eaa5d9d452\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6xsddz"
Nov 01 05:39:44.362375 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:44.362260    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2fd4c9\"/\"agc-cred-17306a2b8c16\""
Nov 01 05:39:44.362375 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:44.362279    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2fd4c9\"/\"kube-root-ca.crt\""
Nov 01 05:39:44.362375 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:44.362260    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2fd4c9\"/\"agc-cred-f8c698a71031\""
Nov 01 05:39:44.362375 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:44.362255    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2fd4c9\"/\"payload-processing-external-model-e2e--928c4e32-dockercfg-r25vt\""
Nov 01 05:39:44.362375 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:44.362294    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2fd4c9\"/\"agc-cred-e326123de79d\""
Nov 01 05:39:44.362375 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:44.362340    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2fd4c9\"/\"routing-overlay\""
Nov 01 05:39:44.364946 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:44.364924    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6xsddz"]
Nov 01 05:39:44.426812 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:44.426781    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a08eb536-6465b89f5b-xgr2d"]
Nov 01 05:39:44.430748 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:44.430730    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a08eb536-6465b89f5b-xgr2d"
Nov 01 05:39:44.433267 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:44.433239    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/52a16023-0c79-474b-ae68-42eaa5d9d452-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6xsddz\" (UID: \"52a16023-0c79-474b-ae68-42eaa5d9d452\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6xsddz"
Nov 01 05:39:44.433393 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:44.433300    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5lnv\" (UniqueName: \"kubernetes.io/projected/52a16023-0c79-474b-ae68-42eaa5d9d452-kube-api-access-z5lnv\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6xsddz\" (UID: \"52a16023-0c79-474b-ae68-42eaa5d9d452\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6xsddz"
Nov 01 05:39:44.433393 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:44.433333    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52a16023-0c79-474b-ae68-42eaa5d9d452-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6xsddz\" (UID: \"52a16023-0c79-474b-ae68-42eaa5d9d452\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6xsddz"
Nov 01 05:39:44.433393 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:44.433387    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0edbadf-884c-409f-9ce4-5b4182df2775-metrics-certs\") pod \"maas-api-e2e-mt-a08eb536-6465b89f5b-xgr2d\" (UID: \"e0edbadf-884c-409f-9ce4-5b4182df2775\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a08eb536-6465b89f5b-xgr2d"
Nov 01 05:39:44.433696 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:44.433419    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0edbadf-884c-409f-9ce4-5b4182df2775-maas-api-tls\") pod \"maas-api-e2e-mt-a08eb536-6465b89f5b-xgr2d\" (UID: \"e0edbadf-884c-409f-9ce4-5b4182df2775\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a08eb536-6465b89f5b-xgr2d"
Nov 01 05:39:44.433696 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:44.433560    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/52a16023-0c79-474b-ae68-42eaa5d9d452-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6xsddz\" (UID: \"52a16023-0c79-474b-ae68-42eaa5d9d452\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6xsddz"
Nov 01 05:39:44.433696 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:44.433626    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0edbadf-884c-409f-9ce4-5b4182df2775-openshift-service-ca\") pod \"maas-api-e2e-mt-a08eb536-6465b89f5b-xgr2d\" (UID: \"e0edbadf-884c-409f-9ce4-5b4182df2775\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a08eb536-6465b89f5b-xgr2d"
Nov 01 05:39:44.433893 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:44.433733    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhzbq\" (UniqueName: \"kubernetes.io/projected/e0edbadf-884c-409f-9ce4-5b4182df2775-kube-api-access-dhzbq\") pod \"maas-api-e2e-mt-a08eb536-6465b89f5b-xgr2d\" (UID: \"e0edbadf-884c-409f-9ce4-5b4182df2775\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a08eb536-6465b89f5b-xgr2d"
Nov 01 05:39:44.434372 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:44.434343    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/52a16023-0c79-474b-ae68-42eaa5d9d452-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6xsddz\" (UID: \"52a16023-0c79-474b-ae68-42eaa5d9d452\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6xsddz"
Nov 01 05:39:44.434498 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:44.434401    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52a16023-0c79-474b-ae68-42eaa5d9d452-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6xsddz\" (UID: \"52a16023-0c79-474b-ae68-42eaa5d9d452\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6xsddz"
Nov 01 05:39:44.436216 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:44.436194    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/52a16023-0c79-474b-ae68-42eaa5d9d452-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6xsddz\" (UID: \"52a16023-0c79-474b-ae68-42eaa5d9d452\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6xsddz"
Nov 01 05:39:44.501964 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:44.501941    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-a08eb536-serving-cert\""
Nov 01 05:39:44.514521 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:44.514493    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a08eb536-6465b89f5b-xgr2d"]
Nov 01 05:39:44.521495 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:44.520584    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5lnv\" (UniqueName: \"kubernetes.io/projected/52a16023-0c79-474b-ae68-42eaa5d9d452-kube-api-access-z5lnv\") pod \"payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6xsddz\" (UID: \"52a16023-0c79-474b-ae68-42eaa5d9d452\") " pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6xsddz"
Nov 01 05:39:44.534877 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:44.534849    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0edbadf-884c-409f-9ce4-5b4182df2775-metrics-certs\") pod \"maas-api-e2e-mt-a08eb536-6465b89f5b-xgr2d\" (UID: \"e0edbadf-884c-409f-9ce4-5b4182df2775\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a08eb536-6465b89f5b-xgr2d"
Nov 01 05:39:44.534987 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:44.534899    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0edbadf-884c-409f-9ce4-5b4182df2775-maas-api-tls\") pod \"maas-api-e2e-mt-a08eb536-6465b89f5b-xgr2d\" (UID: \"e0edbadf-884c-409f-9ce4-5b4182df2775\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a08eb536-6465b89f5b-xgr2d"
Nov 01 05:39:44.535051 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:44.534985    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0edbadf-884c-409f-9ce4-5b4182df2775-openshift-service-ca\") pod \"maas-api-e2e-mt-a08eb536-6465b89f5b-xgr2d\" (UID: \"e0edbadf-884c-409f-9ce4-5b4182df2775\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a08eb536-6465b89f5b-xgr2d"
Nov 01 05:39:44.535128 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:44.535110    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhzbq\" (UniqueName: \"kubernetes.io/projected/e0edbadf-884c-409f-9ce4-5b4182df2775-kube-api-access-dhzbq\") pod \"maas-api-e2e-mt-a08eb536-6465b89f5b-xgr2d\" (UID: \"e0edbadf-884c-409f-9ce4-5b4182df2775\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a08eb536-6465b89f5b-xgr2d"
Nov 01 05:39:44.535189 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:39:44.535133    5689 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-a08eb536-serving-cert: secret "maas-api-e2e-mt-a08eb536-serving-cert" not found
Nov 01 05:39:44.535246 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:39:44.535197    5689 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0edbadf-884c-409f-9ce4-5b4182df2775-maas-api-tls podName:e0edbadf-884c-409f-9ce4-5b4182df2775 nodeName:}" failed. No retries permitted until 2026-11-01 05:39:45.035177743 +0000 UTC m=+2163.105974451 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e0edbadf-884c-409f-9ce4-5b4182df2775-maas-api-tls") pod "maas-api-e2e-mt-a08eb536-6465b89f5b-xgr2d" (UID: "e0edbadf-884c-409f-9ce4-5b4182df2775") : secret "maas-api-e2e-mt-a08eb536-serving-cert" not found
Nov 01 05:39:44.536021 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:44.535999    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0edbadf-884c-409f-9ce4-5b4182df2775-openshift-service-ca\") pod \"maas-api-e2e-mt-a08eb536-6465b89f5b-xgr2d\" (UID: \"e0edbadf-884c-409f-9ce4-5b4182df2775\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a08eb536-6465b89f5b-xgr2d"
Nov 01 05:39:44.537369 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:44.537351    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0edbadf-884c-409f-9ce4-5b4182df2775-metrics-certs\") pod \"maas-api-e2e-mt-a08eb536-6465b89f5b-xgr2d\" (UID: \"e0edbadf-884c-409f-9ce4-5b4182df2775\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a08eb536-6465b89f5b-xgr2d"
Nov 01 05:39:44.574049 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:44.574022    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6xsddz"
Nov 01 05:39:44.599288 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:44.599266    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhzbq\" (UniqueName: \"kubernetes.io/projected/e0edbadf-884c-409f-9ce4-5b4182df2775-kube-api-access-dhzbq\") pod \"maas-api-e2e-mt-a08eb536-6465b89f5b-xgr2d\" (UID: \"e0edbadf-884c-409f-9ce4-5b4182df2775\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a08eb536-6465b89f5b-xgr2d"
Nov 01 05:39:44.706989 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:44.706915    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:39:44.948084 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:44.948051    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:39:44.970363 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:44.970336    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6xsddz"]
Nov 01 05:39:44.977057 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:39:44.977032    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52a16023_0c79_474b_ae68_42eaa5d9d452.slice/crio-e0e590ac2ade4a124e8ec3809d3eb79e31303fe05f62b977f110dee9cc118cbd WatchSource:0}: Error finding container e0e590ac2ade4a124e8ec3809d3eb79e31303fe05f62b977f110dee9cc118cbd: Status 404 returned error can't find the container with id e0e590ac2ade4a124e8ec3809d3eb79e31303fe05f62b977f110dee9cc118cbd
Nov 01 05:39:45.043848 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:45.043826    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0edbadf-884c-409f-9ce4-5b4182df2775-maas-api-tls\") pod \"maas-api-e2e-mt-a08eb536-6465b89f5b-xgr2d\" (UID: \"e0edbadf-884c-409f-9ce4-5b4182df2775\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a08eb536-6465b89f5b-xgr2d"
Nov 01 05:39:45.043976 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:39:45.043959    5689 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-a08eb536-serving-cert: secret "maas-api-e2e-mt-a08eb536-serving-cert" not found
Nov 01 05:39:45.044023 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:39:45.044015    5689 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0edbadf-884c-409f-9ce4-5b4182df2775-maas-api-tls podName:e0edbadf-884c-409f-9ce4-5b4182df2775 nodeName:}" failed. No retries permitted until 2026-11-01 05:39:46.044002363 +0000 UTC m=+2164.114799061 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e0edbadf-884c-409f-9ce4-5b4182df2775-maas-api-tls") pod "maas-api-e2e-mt-a08eb536-6465b89f5b-xgr2d" (UID: "e0edbadf-884c-409f-9ce4-5b4182df2775") : secret "maas-api-e2e-mt-a08eb536-serving-cert" not found
Nov 01 05:39:45.213744 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:45.213665    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:39:45.785423 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:45.785389    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6xsddz" event={"ID":"52a16023-0c79-474b-ae68-42eaa5d9d452","Type":"ContainerStarted","Data":"320415eb37b9d0ff3d4d7997672fc3503ac1951d81c3fd3e3788b78d5e642f25"}
Nov 01 05:39:45.785613 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:45.785447    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6xsddz" event={"ID":"52a16023-0c79-474b-ae68-42eaa5d9d452","Type":"ContainerStarted","Data":"e0e590ac2ade4a124e8ec3809d3eb79e31303fe05f62b977f110dee9cc118cbd"}
Nov 01 05:39:45.785613 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:45.785604    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6xsddz"
Nov 01 05:39:45.903136 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:45.903087    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6xsddz" podStartSLOduration=1.9030738120000001 podStartE2EDuration="1.903073812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:39:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:39:45.901734335 +0000 UTC m=+2163.972531055" watchObservedRunningTime="2026-11-01 05:39:45.903073812 +0000 UTC m=+2163.973870532"
Nov 01 05:39:46.056152 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:46.056081    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0edbadf-884c-409f-9ce4-5b4182df2775-maas-api-tls\") pod \"maas-api-e2e-mt-a08eb536-6465b89f5b-xgr2d\" (UID: \"e0edbadf-884c-409f-9ce4-5b4182df2775\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a08eb536-6465b89f5b-xgr2d"
Nov 01 05:39:46.058605 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:46.058578    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0edbadf-884c-409f-9ce4-5b4182df2775-maas-api-tls\") pod \"maas-api-e2e-mt-a08eb536-6465b89f5b-xgr2d\" (UID: \"e0edbadf-884c-409f-9ce4-5b4182df2775\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a08eb536-6465b89f5b-xgr2d"
Nov 01 05:39:46.254830 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:46.254800    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a08eb536-6465b89f5b-xgr2d"
Nov 01 05:39:46.631149 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:46.631005    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a08eb536-6465b89f5b-xgr2d"]
Nov 01 05:39:46.633929 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:39:46.633891    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0edbadf_884c_409f_9ce4_5b4182df2775.slice/crio-4e0088f658bd0dc590301fca13be41822e91be3683b36b46501eb1d06049973a WatchSource:0}: Error finding container 4e0088f658bd0dc590301fca13be41822e91be3683b36b46501eb1d06049973a: Status 404 returned error can't find the container with id 4e0088f658bd0dc590301fca13be41822e91be3683b36b46501eb1d06049973a
Nov 01 05:39:46.635555 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:46.635538    5689 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 05:39:46.791948 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:46.791915    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a08eb536-6465b89f5b-xgr2d" event={"ID":"e0edbadf-884c-409f-9ce4-5b4182df2775","Type":"ContainerStarted","Data":"4e0088f658bd0dc590301fca13be41822e91be3683b36b46501eb1d06049973a"}
Nov 01 05:39:47.799399 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:47.799319    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a08eb536-6465b89f5b-xgr2d" event={"ID":"e0edbadf-884c-409f-9ce4-5b4182df2775","Type":"ContainerStarted","Data":"528634dfff97f941aa8f639d67d8a31718e9c711a0ed1a0e677d0e7b63bf9c97"}
Nov 01 05:39:47.799807 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:47.799495    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a08eb536-6465b89f5b-xgr2d"
Nov 01 05:39:47.916503 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:47.916433    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a08eb536-6465b89f5b-xgr2d" podStartSLOduration=3.106354513 podStartE2EDuration="3.916418209s" totalImagesPullingTime="810.063696ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:39:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:39:47.915289749 +0000 UTC m=+2165.986086470" watchObservedRunningTime="2026-11-01 05:39:47.916418209 +0000 UTC m=+2165.987214929"
Nov 01 05:39:48.567299 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:48.567239    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:39:48.749204 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:48.749167    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:39:49.694421 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:49.694387    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:39:50.143265 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:50.143161    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:39:50.330201 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:50.330163    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:39:50.589824 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:50.589792    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:39:51.669209 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:51.669168    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:39:51.866735 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:51.866700    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:39:53.811941 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:53.811910    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a08eb536-6465b89f5b-xgr2d"
Nov 01 05:39:53.981906 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:53.981870    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:39:54.135265 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:54.134824    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6xsddz"]
Nov 01 05:39:54.139866 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:54.139835    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6xsddz" podUID="52a16023-0c79-474b-ae68-42eaa5d9d452" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 05:39:54.622580 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:54.622546    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:39:59.136533 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:59.136492    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6xsddz" podUID="52a16023-0c79-474b-ae68-42eaa5d9d452" containerName="payload-processing" containerID="cri-o://320415eb37b9d0ff3d4d7997672fc3503ac1951d81c3fd3e3788b78d5e642f25" gracePeriod=25
Nov 01 05:39:59.402402 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:59.402375    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6xsddz"
Nov 01 05:39:59.435468 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:59.435424    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:39:59.590669 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:59.590640    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52a16023-0c79-474b-ae68-42eaa5d9d452-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52a16023-0c79-474b-ae68-42eaa5d9d452-plugins-config-volume\") pod \"52a16023-0c79-474b-ae68-42eaa5d9d452\" (UID: \"52a16023-0c79-474b-ae68-42eaa5d9d452\") "
Nov 01 05:39:59.590865 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:59.590811    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52a16023-0c79-474b-ae68-42eaa5d9d452-provider-credentials\" (UniqueName: \"kubernetes.io/projected/52a16023-0c79-474b-ae68-42eaa5d9d452-provider-credentials\") pod \"52a16023-0c79-474b-ae68-42eaa5d9d452\" (UID: \"52a16023-0c79-474b-ae68-42eaa5d9d452\") "
Nov 01 05:39:59.590932 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:59.590862    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52a16023-0c79-474b-ae68-42eaa5d9d452-kube-api-access-z5lnv\" (UniqueName: \"kubernetes.io/projected/52a16023-0c79-474b-ae68-42eaa5d9d452-kube-api-access-z5lnv\") pod \"52a16023-0c79-474b-ae68-42eaa5d9d452\" (UID: \"52a16023-0c79-474b-ae68-42eaa5d9d452\") "
Nov 01 05:39:59.590987 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:59.590935    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52a16023-0c79-474b-ae68-42eaa5d9d452-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/52a16023-0c79-474b-ae68-42eaa5d9d452-routing-overlay-volume\") pod \"52a16023-0c79-474b-ae68-42eaa5d9d452\" (UID: \"52a16023-0c79-474b-ae68-42eaa5d9d452\") "
Nov 01 05:39:59.591068 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:59.591044    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52a16023-0c79-474b-ae68-42eaa5d9d452-plugins-config-volume" pod "52a16023-0c79-474b-ae68-42eaa5d9d452" (UID: "52a16023-0c79-474b-ae68-42eaa5d9d452"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:39:59.591315 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:59.591293    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52a16023-0c79-474b-ae68-42eaa5d9d452-routing-overlay-volume" pod "52a16023-0c79-474b-ae68-42eaa5d9d452" (UID: "52a16023-0c79-474b-ae68-42eaa5d9d452"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:39:59.591402 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:59.591383    5689 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/52a16023-0c79-474b-ae68-42eaa5d9d452-routing-overlay-volume\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:39:59.591402 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:59.591396    5689 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52a16023-0c79-474b-ae68-42eaa5d9d452-plugins-config-volume\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:39:59.593094 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:59.593074    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52a16023-0c79-474b-ae68-42eaa5d9d452-provider-credentials" pod "52a16023-0c79-474b-ae68-42eaa5d9d452" (UID: "52a16023-0c79-474b-ae68-42eaa5d9d452"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:39:59.593148 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:59.593103    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52a16023-0c79-474b-ae68-42eaa5d9d452-kube-api-access-z5lnv" pod "52a16023-0c79-474b-ae68-42eaa5d9d452" (UID: "52a16023-0c79-474b-ae68-42eaa5d9d452"). InnerVolumeSpecName "kube-api-access-z5lnv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:39:59.692697 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:59.692612    5689 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/52a16023-0c79-474b-ae68-42eaa5d9d452-provider-credentials\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:39:59.692697 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:59.692641    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5lnv\" (UniqueName: \"kubernetes.io/projected/52a16023-0c79-474b-ae68-42eaa5d9d452-kube-api-access-z5lnv\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:39:59.856412 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:59.856378    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:39:59.871525 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:59.871491    5689 generic.go:356] "Generic (PLEG): container finished" podID="52a16023-0c79-474b-ae68-42eaa5d9d452" containerID="320415eb37b9d0ff3d4d7997672fc3503ac1951d81c3fd3e3788b78d5e642f25" exitCode=0
Nov 01 05:39:59.871663 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:59.871532    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6xsddz" event={"ID":"52a16023-0c79-474b-ae68-42eaa5d9d452","Type":"ContainerDied","Data":"320415eb37b9d0ff3d4d7997672fc3503ac1951d81c3fd3e3788b78d5e642f25"}
Nov 01 05:39:59.871663 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:59.871607    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6xsddz" event={"ID":"52a16023-0c79-474b-ae68-42eaa5d9d452","Type":"ContainerDied","Data":"e0e590ac2ade4a124e8ec3809d3eb79e31303fe05f62b977f110dee9cc118cbd"}
Nov 01 05:39:59.871663 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:59.871538    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6xsddz"
Nov 01 05:39:59.871663 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:59.871631    5689 scope.go:122] "RemoveContainer" containerID="320415eb37b9d0ff3d4d7997672fc3503ac1951d81c3fd3e3788b78d5e642f25"
Nov 01 05:39:59.911706 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:59.909684    5689 scope.go:122] "RemoveContainer" containerID="320415eb37b9d0ff3d4d7997672fc3503ac1951d81c3fd3e3788b78d5e642f25"
Nov 01 05:39:59.911706 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:39:59.910178    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"320415eb37b9d0ff3d4d7997672fc3503ac1951d81c3fd3e3788b78d5e642f25\": container with ID starting with 320415eb37b9d0ff3d4d7997672fc3503ac1951d81c3fd3e3788b78d5e642f25 not found: ID does not exist" containerID="320415eb37b9d0ff3d4d7997672fc3503ac1951d81c3fd3e3788b78d5e642f25"
Nov 01 05:39:59.911706 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:39:59.910214    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"320415eb37b9d0ff3d4d7997672fc3503ac1951d81c3fd3e3788b78d5e642f25"} err="failed to get container status \"320415eb37b9d0ff3d4d7997672fc3503ac1951d81c3fd3e3788b78d5e642f25\": rpc error: code = NotFound desc = could not find container \"320415eb37b9d0ff3d4d7997672fc3503ac1951d81c3fd3e3788b78d5e642f25\": container with ID starting with 320415eb37b9d0ff3d4d7997672fc3503ac1951d81c3fd3e3788b78d5e642f25 not found: ID does not exist"
Nov 01 05:40:00.060169 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:40:00.060134    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6xsddz"]
Nov 01 05:40:00.068039 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:40:00.068008    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-2fd4c9/payload-processing-external-model-e2e-extmod-b-2fd4c9-85c6xsddz"]
Nov 01 05:40:00.070532 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:40:00.070503    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:40:02.067794 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:40:02.067759    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52a16023-0c79-474b-ae68-42eaa5d9d452" path="/var/lib/kubelet/pods/52a16023-0c79-474b-ae68-42eaa5d9d452/volumes"
Nov 01 05:40:04.505835 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:40:04.505804    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:40:06.027273 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:40:06.027237    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:40:09.656563 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:40:09.656530    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:40:14.501975 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:40:14.501938    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:40:15.175876 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:40:15.175846    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:40:15.350783 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:40:15.350751    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:40:17.562821 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:40:17.562788    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4cbcb62b-849d7dcbc9-vjtls"]
Nov 01 05:40:17.669733 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:40:17.669703    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4cbcb62b-5d57bdcc59-vpvfg"]
Nov 01 05:40:17.963047 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:40:17.962987    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:40:18.157482 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:40:18.154072    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:40:19.315022 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:40:19.314987    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:40:22.563737 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:40:22.563698    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-4cbcb62b-849d7dcbc9-vjtls" podUID="e2f0666e-d262-4206-ae1f-b4feeeb56c0d" containerName="payload-processing" containerID="cri-o://710a8ef6101b070518de8fb0a3d549a4448f268a75495d290ac4861c81d3afdd" gracePeriod=25
Nov 01 05:40:22.670132 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:40:22.670093    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-4cbcb62b-5d57bdcc59-vpvfg" podUID="1b449f8f-a42d-4645-a555-3c5727026a47" containerName="payload-pre-processing" containerID="cri-o://57712c15953579e62469c06c2eabaa0f7727455ab84100ca4efa9ba08f71162c" gracePeriod=25
Nov 01 05:40:22.833204 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:40:22.833138    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4cbcb62b-849d7dcbc9-vjtls"
Nov 01 05:40:22.894982 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:40:22.894951    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2f0666e-d262-4206-ae1f-b4feeeb56c0d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2f0666e-d262-4206-ae1f-b4feeeb56c0d-plugins-config-volume\") pod \"e2f0666e-d262-4206-ae1f-b4feeeb56c0d\" (UID: \"e2f0666e-d262-4206-ae1f-b4feeeb56c0d\") "
Nov 01 05:40:22.895168 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:40:22.895145    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2f0666e-d262-4206-ae1f-b4feeeb56c0d-kube-api-access-dmwrb\" (UniqueName: \"kubernetes.io/projected/e2f0666e-d262-4206-ae1f-b4feeeb56c0d-kube-api-access-dmwrb\") pod \"e2f0666e-d262-4206-ae1f-b4feeeb56c0d\" (UID: \"e2f0666e-d262-4206-ae1f-b4feeeb56c0d\") "
Nov 01 05:40:22.895395 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:40:22.895347    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2f0666e-d262-4206-ae1f-b4feeeb56c0d-plugins-config-volume" pod "e2f0666e-d262-4206-ae1f-b4feeeb56c0d" (UID: "e2f0666e-d262-4206-ae1f-b4feeeb56c0d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:40:22.895701 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:40:22.895678    5689 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2f0666e-d262-4206-ae1f-b4feeeb56c0d-plugins-config-volume\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:40:22.898079 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:40:22.898053    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2f0666e-d262-4206-ae1f-b4feeeb56c0d-kube-api-access-dmwrb" pod "e2f0666e-d262-4206-ae1f-b4feeeb56c0d" (UID: "e2f0666e-d262-4206-ae1f-b4feeeb56c0d"). InnerVolumeSpecName "kube-api-access-dmwrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:40:22.927795 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:40:22.927775    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4cbcb62b-5d57bdcc59-vpvfg"
Nov 01 05:40:22.996166 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:40:22.996136    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b449f8f-a42d-4645-a555-3c5727026a47-kube-api-access-2lt8m\" (UniqueName: \"kubernetes.io/projected/1b449f8f-a42d-4645-a555-3c5727026a47-kube-api-access-2lt8m\") pod \"1b449f8f-a42d-4645-a555-3c5727026a47\" (UID: \"1b449f8f-a42d-4645-a555-3c5727026a47\") "
Nov 01 05:40:22.996371 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:40:22.996357    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b449f8f-a42d-4645-a555-3c5727026a47-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b449f8f-a42d-4645-a555-3c5727026a47-plugins-config-volume\") pod \"1b449f8f-a42d-4645-a555-3c5727026a47\" (UID: \"1b449f8f-a42d-4645-a555-3c5727026a47\") "
Nov 01 05:40:22.996799 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:40:22.996770    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b449f8f-a42d-4645-a555-3c5727026a47-plugins-config-volume" pod "1b449f8f-a42d-4645-a555-3c5727026a47" (UID: "1b449f8f-a42d-4645-a555-3c5727026a47"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:40:22.996908 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:40:22.996776    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmwrb\" (UniqueName: \"kubernetes.io/projected/e2f0666e-d262-4206-ae1f-b4feeeb56c0d-kube-api-access-dmwrb\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:40:22.998582 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:40:22.998558    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b449f8f-a42d-4645-a555-3c5727026a47-kube-api-access-2lt8m" pod "1b449f8f-a42d-4645-a555-3c5727026a47" (UID: "1b449f8f-a42d-4645-a555-3c5727026a47"). InnerVolumeSpecName "kube-api-access-2lt8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:40:23.008749 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:40:23.008727    5689 generic.go:356] "Generic (PLEG): container finished" podID="e2f0666e-d262-4206-ae1f-b4feeeb56c0d" containerID="710a8ef6101b070518de8fb0a3d549a4448f268a75495d290ac4861c81d3afdd" exitCode=0
Nov 01 05:40:23.008914 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:40:23.008770    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4cbcb62b-849d7dcbc9-vjtls"
Nov 01 05:40:23.008914 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:40:23.008808    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4cbcb62b-849d7dcbc9-vjtls" event={"ID":"e2f0666e-d262-4206-ae1f-b4feeeb56c0d","Type":"ContainerDied","Data":"710a8ef6101b070518de8fb0a3d549a4448f268a75495d290ac4861c81d3afdd"}
Nov 01 05:40:23.008914 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:40:23.008907    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4cbcb62b-849d7dcbc9-vjtls" event={"ID":"e2f0666e-d262-4206-ae1f-b4feeeb56c0d","Type":"ContainerDied","Data":"01ee885431540a2310ff89f14d9eb5a9ccb4cafc497fb47bdb617bbdbe3f6468"}
Nov 01 05:40:23.009038 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:40:23.008926    5689 scope.go:122] "RemoveContainer" containerID="710a8ef6101b070518de8fb0a3d549a4448f268a75495d290ac4861c81d3afdd"
Nov 01 05:40:23.010195 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:40:23.010174    5689 generic.go:356] "Generic (PLEG): container finished" podID="1b449f8f-a42d-4645-a555-3c5727026a47" containerID="57712c15953579e62469c06c2eabaa0f7727455ab84100ca4efa9ba08f71162c" exitCode=0
Nov 01 05:40:23.010279 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:40:23.010202    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4cbcb62b-5d57bdcc59-vpvfg" event={"ID":"1b449f8f-a42d-4645-a555-3c5727026a47","Type":"ContainerDied","Data":"57712c15953579e62469c06c2eabaa0f7727455ab84100ca4efa9ba08f71162c"}
Nov 01 05:40:23.010279 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:40:23.010214    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4cbcb62b-5d57bdcc59-vpvfg"
Nov 01 05:40:23.010279 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:40:23.010247    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4cbcb62b-5d57bdcc59-vpvfg" event={"ID":"1b449f8f-a42d-4645-a555-3c5727026a47","Type":"ContainerDied","Data":"4710fd3f6304abdda499df2c5faf080055c9795b1a5f95a02bc29c5eefb437e0"}
Nov 01 05:40:23.020666 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:40:23.020644    5689 scope.go:122] "RemoveContainer" containerID="710a8ef6101b070518de8fb0a3d549a4448f268a75495d290ac4861c81d3afdd"
Nov 01 05:40:23.026847 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:40:23.026820    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"710a8ef6101b070518de8fb0a3d549a4448f268a75495d290ac4861c81d3afdd\": container with ID starting with 710a8ef6101b070518de8fb0a3d549a4448f268a75495d290ac4861c81d3afdd not found: ID does not exist" containerID="710a8ef6101b070518de8fb0a3d549a4448f268a75495d290ac4861c81d3afdd"
Nov 01 05:40:23.026950 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:40:23.026855    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"710a8ef6101b070518de8fb0a3d549a4448f268a75495d290ac4861c81d3afdd"} err="failed to get container status \"710a8ef6101b070518de8fb0a3d549a4448f268a75495d290ac4861c81d3afdd\": rpc error: code = NotFound desc = could not find container \"710a8ef6101b070518de8fb0a3d549a4448f268a75495d290ac4861c81d3afdd\": container with ID starting with 710a8ef6101b070518de8fb0a3d549a4448f268a75495d290ac4861c81d3afdd not found: ID does not exist"
Nov 01 05:40:23.026950 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:40:23.026897    5689 scope.go:122] "RemoveContainer" containerID="57712c15953579e62469c06c2eabaa0f7727455ab84100ca4efa9ba08f71162c"
Nov 01 05:40:23.036913 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:40:23.036894    5689 scope.go:122] "RemoveContainer" containerID="57712c15953579e62469c06c2eabaa0f7727455ab84100ca4efa9ba08f71162c"
Nov 01 05:40:23.037183 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:40:23.037161    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57712c15953579e62469c06c2eabaa0f7727455ab84100ca4efa9ba08f71162c\": container with ID starting with 57712c15953579e62469c06c2eabaa0f7727455ab84100ca4efa9ba08f71162c not found: ID does not exist" containerID="57712c15953579e62469c06c2eabaa0f7727455ab84100ca4efa9ba08f71162c"
Nov 01 05:40:23.037263 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:40:23.037192    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57712c15953579e62469c06c2eabaa0f7727455ab84100ca4efa9ba08f71162c"} err="failed to get container status \"57712c15953579e62469c06c2eabaa0f7727455ab84100ca4efa9ba08f71162c\": rpc error: code = NotFound desc = could not find container \"57712c15953579e62469c06c2eabaa0f7727455ab84100ca4efa9ba08f71162c\": container with ID starting with 57712c15953579e62469c06c2eabaa0f7727455ab84100ca4efa9ba08f71162c not found: ID does not exist"
Nov 01 05:40:23.098007 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:40:23.097933    5689 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b449f8f-a42d-4645-a555-3c5727026a47-plugins-config-volume\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:40:23.098007 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:40:23.097957    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lt8m\" (UniqueName: \"kubernetes.io/projected/1b449f8f-a42d-4645-a555-3c5727026a47-kube-api-access-2lt8m\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:40:23.173698 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:40:23.173671    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4cbcb62b-5d57bdcc59-vpvfg"]
Nov 01 05:40:23.178038 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:40:23.178017    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4cbcb62b-5d57bdcc59-vpvfg"]
Nov 01 05:40:23.285582 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:40:23.285556    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4cbcb62b-849d7dcbc9-vjtls"]
Nov 01 05:40:23.289685 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:40:23.289661    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4cbcb62b-849d7dcbc9-vjtls"]
Nov 01 05:40:24.067093 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:40:24.067059    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b449f8f-a42d-4645-a555-3c5727026a47" path="/var/lib/kubelet/pods/1b449f8f-a42d-4645-a555-3c5727026a47/volumes"
Nov 01 05:40:24.067525 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:40:24.067509    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2f0666e-d262-4206-ae1f-b4feeeb56c0d" path="/var/lib/kubelet/pods/e2f0666e-d262-4206-ae1f-b4feeeb56c0d/volumes"
Nov 01 05:40:24.283850 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:40:24.283808    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:40:26.596595 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:40:26.596563    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:40:29.162168 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:40:29.162134    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:40:30.069279 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:40:30.069247    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:40:31.932232 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:40:31.932186    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:40:32.545438 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:40:32.543916    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:40:32.956277 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:40:32.956246    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:40:34.064226 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:40:34.064195    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:40:36.406123 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:40:36.406094    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:40:36.712100 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:40:36.712000    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:40:36.909631 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:40:36.909597    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:40:38.603476 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:40:38.587038    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:40:39.277445 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:40:39.277417    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:40:39.458213 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:40:39.458180    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:40:39.659814 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:40:39.659778    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:40:39.836699 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:40:39.836667    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:40:44.558270 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:40:44.558238    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:40:49.326124 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:40:49.326090    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:40:50.418936 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:40:50.418895    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:40:51.314120 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:40:51.314086    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:40:51.491215 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:40:51.491177    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:40:51.955167 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:40:51.955133    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:40:53.800049 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:40:53.800010    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:40:55.476137 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:40:55.476098    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:40:55.727766 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:40:55.727679    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:40:55.904708 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:40:55.904666    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:40:57.462971 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:40:57.462936    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664"]
Nov 01 05:40:57.463386 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:40:57.463118    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664" podUID="9a009de5-76be-4db1-adec-efb9246d4ddf" containerName="istio-proxy" containerID="cri-o://615b7abd04ba4d081fe29d39a77bbe982cd8a97e4ca68d46e26c856b8c2b94b3" gracePeriod=30
Nov 01 05:40:57.501373 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:40:57.501347    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:40:58.070118 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:40:58.070090    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:40:58.547086 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:40:58.547050    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:40:58.730959 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:40:58.730928    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:40:58.908914 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:40:58.908845    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:41:00.069708 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:41:00.069676    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:41:00.896297 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:00.896264    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:41:01.854276 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:01.854244    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:41:02.045468 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:02.045411    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:41:02.228244 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:02.228214    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:41:02.737242 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:02.737188    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664"
Nov 01 05:41:02.843090 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:02.843064    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a009de5-76be-4db1-adec-efb9246d4ddf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a009de5-76be-4db1-adec-efb9246d4ddf-istiod-ca-cert\") pod \"9a009de5-76be-4db1-adec-efb9246d4ddf\" (UID: \"9a009de5-76be-4db1-adec-efb9246d4ddf\") "
Nov 01 05:41:02.843257 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:02.843120    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a009de5-76be-4db1-adec-efb9246d4ddf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a009de5-76be-4db1-adec-efb9246d4ddf-credential-socket\") pod \"9a009de5-76be-4db1-adec-efb9246d4ddf\" (UID: \"9a009de5-76be-4db1-adec-efb9246d4ddf\") "
Nov 01 05:41:02.843257 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:02.843153    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a009de5-76be-4db1-adec-efb9246d4ddf-kube-api-access-pc8lp\" (UniqueName: \"kubernetes.io/projected/9a009de5-76be-4db1-adec-efb9246d4ddf-kube-api-access-pc8lp\") pod \"9a009de5-76be-4db1-adec-efb9246d4ddf\" (UID: \"9a009de5-76be-4db1-adec-efb9246d4ddf\") "
Nov 01 05:41:02.843257 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:02.843169    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a009de5-76be-4db1-adec-efb9246d4ddf-istio-token\" (UniqueName: \"kubernetes.io/projected/9a009de5-76be-4db1-adec-efb9246d4ddf-istio-token\") pod \"9a009de5-76be-4db1-adec-efb9246d4ddf\" (UID: \"9a009de5-76be-4db1-adec-efb9246d4ddf\") "
Nov 01 05:41:02.843257 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:02.843212    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a009de5-76be-4db1-adec-efb9246d4ddf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a009de5-76be-4db1-adec-efb9246d4ddf-istio-envoy\") pod \"9a009de5-76be-4db1-adec-efb9246d4ddf\" (UID: \"9a009de5-76be-4db1-adec-efb9246d4ddf\") "
Nov 01 05:41:02.843257 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:02.843231    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a009de5-76be-4db1-adec-efb9246d4ddf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a009de5-76be-4db1-adec-efb9246d4ddf-workload-certs\") pod \"9a009de5-76be-4db1-adec-efb9246d4ddf\" (UID: \"9a009de5-76be-4db1-adec-efb9246d4ddf\") "
Nov 01 05:41:02.843541 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:02.843314    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a009de5-76be-4db1-adec-efb9246d4ddf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a009de5-76be-4db1-adec-efb9246d4ddf-istio-data\") pod \"9a009de5-76be-4db1-adec-efb9246d4ddf\" (UID: \"9a009de5-76be-4db1-adec-efb9246d4ddf\") "
Nov 01 05:41:02.843541 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:02.843345    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a009de5-76be-4db1-adec-efb9246d4ddf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a009de5-76be-4db1-adec-efb9246d4ddf-istio-podinfo\") pod \"9a009de5-76be-4db1-adec-efb9246d4ddf\" (UID: \"9a009de5-76be-4db1-adec-efb9246d4ddf\") "
Nov 01 05:41:02.843541 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:02.843423    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a009de5-76be-4db1-adec-efb9246d4ddf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a009de5-76be-4db1-adec-efb9246d4ddf-workload-socket\") pod \"9a009de5-76be-4db1-adec-efb9246d4ddf\" (UID: \"9a009de5-76be-4db1-adec-efb9246d4ddf\") "
Nov 01 05:41:02.843708 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:02.843531    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a009de5-76be-4db1-adec-efb9246d4ddf-credential-socket" pod "9a009de5-76be-4db1-adec-efb9246d4ddf" (UID: "9a009de5-76be-4db1-adec-efb9246d4ddf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:41:02.843708 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:02.843539    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a009de5-76be-4db1-adec-efb9246d4ddf-istiod-ca-cert" pod "9a009de5-76be-4db1-adec-efb9246d4ddf" (UID: "9a009de5-76be-4db1-adec-efb9246d4ddf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:41:02.843823 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:02.843743    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a009de5-76be-4db1-adec-efb9246d4ddf-istio-data" pod "9a009de5-76be-4db1-adec-efb9246d4ddf" (UID: "9a009de5-76be-4db1-adec-efb9246d4ddf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:41:02.844108 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:02.843925    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a009de5-76be-4db1-adec-efb9246d4ddf-workload-socket" pod "9a009de5-76be-4db1-adec-efb9246d4ddf" (UID: "9a009de5-76be-4db1-adec-efb9246d4ddf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:41:02.844108 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:02.843952    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a009de5-76be-4db1-adec-efb9246d4ddf-workload-certs" pod "9a009de5-76be-4db1-adec-efb9246d4ddf" (UID: "9a009de5-76be-4db1-adec-efb9246d4ddf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:41:02.844356 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:02.844334    5689 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a009de5-76be-4db1-adec-efb9246d4ddf-istio-data\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:41:02.844431 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:02.844362    5689 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a009de5-76be-4db1-adec-efb9246d4ddf-workload-socket\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:41:02.844431 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:02.844380    5689 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a009de5-76be-4db1-adec-efb9246d4ddf-istiod-ca-cert\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:41:02.844431 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:02.844395    5689 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a009de5-76be-4db1-adec-efb9246d4ddf-credential-socket\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:41:02.844431 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:02.844410    5689 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a009de5-76be-4db1-adec-efb9246d4ddf-workload-certs\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:41:02.845680 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:02.845648    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a009de5-76be-4db1-adec-efb9246d4ddf-istio-envoy" pod "9a009de5-76be-4db1-adec-efb9246d4ddf" (UID: "9a009de5-76be-4db1-adec-efb9246d4ddf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:41:02.845802 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:02.845758    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a009de5-76be-4db1-adec-efb9246d4ddf-kube-api-access-pc8lp" pod "9a009de5-76be-4db1-adec-efb9246d4ddf" (UID: "9a009de5-76be-4db1-adec-efb9246d4ddf"). InnerVolumeSpecName "kube-api-access-pc8lp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:41:02.845802 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:02.845785    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a009de5-76be-4db1-adec-efb9246d4ddf-istio-podinfo" pod "9a009de5-76be-4db1-adec-efb9246d4ddf" (UID: "9a009de5-76be-4db1-adec-efb9246d4ddf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 05:41:02.845802 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:02.845790    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a009de5-76be-4db1-adec-efb9246d4ddf-istio-token" pod "9a009de5-76be-4db1-adec-efb9246d4ddf" (UID: "9a009de5-76be-4db1-adec-efb9246d4ddf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:41:02.945621 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:02.945591    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pc8lp\" (UniqueName: \"kubernetes.io/projected/9a009de5-76be-4db1-adec-efb9246d4ddf-kube-api-access-pc8lp\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:41:02.945621 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:02.945625    5689 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a009de5-76be-4db1-adec-efb9246d4ddf-istio-token\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:41:02.946016 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:02.945640    5689 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a009de5-76be-4db1-adec-efb9246d4ddf-istio-envoy\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:41:02.946016 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:02.945654    5689 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a009de5-76be-4db1-adec-efb9246d4ddf-istio-podinfo\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:41:03.239786 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:03.239748    5689 generic.go:356] "Generic (PLEG): container finished" podID="9a009de5-76be-4db1-adec-efb9246d4ddf" containerID="615b7abd04ba4d081fe29d39a77bbe982cd8a97e4ca68d46e26c856b8c2b94b3" exitCode=0
Nov 01 05:41:03.239933 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:03.239793    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664"
Nov 01 05:41:03.239933 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:03.239830    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664" event={"ID":"9a009de5-76be-4db1-adec-efb9246d4ddf","Type":"ContainerDied","Data":"615b7abd04ba4d081fe29d39a77bbe982cd8a97e4ca68d46e26c856b8c2b94b3"}
Nov 01 05:41:03.239933 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:03.239876    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664" event={"ID":"9a009de5-76be-4db1-adec-efb9246d4ddf","Type":"ContainerDied","Data":"5ad2a62497190b33b1e20c2db3720de321402ccfcb47a20e8947343ee98993a6"}
Nov 01 05:41:03.239933 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:03.239899    5689 scope.go:122] "RemoveContainer" containerID="615b7abd04ba4d081fe29d39a77bbe982cd8a97e4ca68d46e26c856b8c2b94b3"
Nov 01 05:41:03.252102 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:03.252082    5689 scope.go:122] "RemoveContainer" containerID="fce93fd45f1a7f3f0bbb753a9fee3c2f9ffaab23e93fe6e9021ea2c411352c9c"
Nov 01 05:41:03.261507 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:03.261490    5689 scope.go:122] "RemoveContainer" containerID="615b7abd04ba4d081fe29d39a77bbe982cd8a97e4ca68d46e26c856b8c2b94b3"
Nov 01 05:41:03.261779 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:41:03.261760    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"615b7abd04ba4d081fe29d39a77bbe982cd8a97e4ca68d46e26c856b8c2b94b3\": container with ID starting with 615b7abd04ba4d081fe29d39a77bbe982cd8a97e4ca68d46e26c856b8c2b94b3 not found: ID does not exist" containerID="615b7abd04ba4d081fe29d39a77bbe982cd8a97e4ca68d46e26c856b8c2b94b3"
Nov 01 05:41:03.261839 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:03.261788    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"615b7abd04ba4d081fe29d39a77bbe982cd8a97e4ca68d46e26c856b8c2b94b3"} err="failed to get container status \"615b7abd04ba4d081fe29d39a77bbe982cd8a97e4ca68d46e26c856b8c2b94b3\": rpc error: code = NotFound desc = could not find container \"615b7abd04ba4d081fe29d39a77bbe982cd8a97e4ca68d46e26c856b8c2b94b3\": container with ID starting with 615b7abd04ba4d081fe29d39a77bbe982cd8a97e4ca68d46e26c856b8c2b94b3 not found: ID does not exist"
Nov 01 05:41:03.261839 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:03.261805    5689 scope.go:122] "RemoveContainer" containerID="fce93fd45f1a7f3f0bbb753a9fee3c2f9ffaab23e93fe6e9021ea2c411352c9c"
Nov 01 05:41:03.262048 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:41:03.262029    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fce93fd45f1a7f3f0bbb753a9fee3c2f9ffaab23e93fe6e9021ea2c411352c9c\": container with ID starting with fce93fd45f1a7f3f0bbb753a9fee3c2f9ffaab23e93fe6e9021ea2c411352c9c not found: ID does not exist" containerID="fce93fd45f1a7f3f0bbb753a9fee3c2f9ffaab23e93fe6e9021ea2c411352c9c"
Nov 01 05:41:03.262104 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:03.262054    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fce93fd45f1a7f3f0bbb753a9fee3c2f9ffaab23e93fe6e9021ea2c411352c9c"} err="failed to get container status \"fce93fd45f1a7f3f0bbb753a9fee3c2f9ffaab23e93fe6e9021ea2c411352c9c\": rpc error: code = NotFound desc = could not find container \"fce93fd45f1a7f3f0bbb753a9fee3c2f9ffaab23e93fe6e9021ea2c411352c9c\": container with ID starting with fce93fd45f1a7f3f0bbb753a9fee3c2f9ffaab23e93fe6e9021ea2c411352c9c not found: ID does not exist"
Nov 01 05:41:03.404441 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:03.404406    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664"]
Nov 01 05:41:03.453809 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:03.453734    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-4cbcb62b-openshift-default-7657bb7b8f-tt664"]
Nov 01 05:41:04.067682 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:04.067649    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a009de5-76be-4db1-adec-efb9246d4ddf" path="/var/lib/kubelet/pods/9a009de5-76be-4db1-adec-efb9246d4ddf/volumes"
Nov 01 05:41:05.133747 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:05.133715    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:41:05.313720 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:05.313690    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:41:05.681517 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:05.675075    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ad90c52d-d779bc747-ngwgg"]
Nov 01 05:41:05.683712 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:05.683638    5689 cpu_manager.go:414] "containerMap: removing container" podUID="52a16023-0c79-474b-ae68-42eaa5d9d452" containerName="payload-processing"
Nov 01 05:41:05.687518 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:05.687488    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52a16023-0c79-474b-ae68-42eaa5d9d452" containerName="payload-processing"
Nov 01 05:41:05.687734 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:05.687719    5689 cpu_manager.go:414] "containerMap: removing container" podUID="9a009de5-76be-4db1-adec-efb9246d4ddf" containerName="istio-proxy"
Nov 01 05:41:05.687986 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:05.687813    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a009de5-76be-4db1-adec-efb9246d4ddf" containerName="istio-proxy"
Nov 01 05:41:05.702593 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:05.702558    5689 cpu_manager.go:414] "containerMap: removing container" podUID="1b449f8f-a42d-4645-a555-3c5727026a47" containerName="payload-pre-processing"
Nov 01 05:41:05.702593 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:05.702591    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b449f8f-a42d-4645-a555-3c5727026a47" containerName="payload-pre-processing"
Nov 01 05:41:05.702801 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:05.702629    5689 cpu_manager.go:414] "containerMap: removing container" podUID="e2f0666e-d262-4206-ae1f-b4feeeb56c0d" containerName="payload-processing"
Nov 01 05:41:05.702801 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:05.702638    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2f0666e-d262-4206-ae1f-b4feeeb56c0d" containerName="payload-processing"
Nov 01 05:41:05.702916 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:05.702900    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b449f8f-a42d-4645-a555-3c5727026a47" containerName="payload-pre-processing"
Nov 01 05:41:05.702962 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:05.702925    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2f0666e-d262-4206-ae1f-b4feeeb56c0d" containerName="payload-processing"
Nov 01 05:41:05.702962 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:05.702935    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a009de5-76be-4db1-adec-efb9246d4ddf" containerName="istio-proxy"
Nov 01 05:41:05.702962 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:05.702952    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="52a16023-0c79-474b-ae68-42eaa5d9d452" containerName="payload-processing"
Nov 01 05:41:05.703062 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:05.702967    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a009de5-76be-4db1-adec-efb9246d4ddf" containerName="istio-proxy"
Nov 01 05:41:05.706761 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:05.706737    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ad90c52d-d779bc747-ngwgg"
Nov 01 05:41:05.731380 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:05.731338    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ad90c52d-d779bc747-ngwgg"]
Nov 01 05:41:05.777308 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:05.777273    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-ad90c52d\""
Nov 01 05:41:05.777308 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:05.777285    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-ad90c52d-dockercfg-7wz44\""
Nov 01 05:41:05.792046 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:05.792021    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9118c67d-7bca-44ad-afc8-102cb132e193-plugins-config-volume\") pod \"payload-processing-e2e-ait-ad90c52d-d779bc747-ngwgg\" (UID: \"9118c67d-7bca-44ad-afc8-102cb132e193\") " pod="openshift-ingress/payload-processing-e2e-ait-ad90c52d-d779bc747-ngwgg"
Nov 01 05:41:05.792219 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:05.792063    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwfgj\" (UniqueName: \"kubernetes.io/projected/9118c67d-7bca-44ad-afc8-102cb132e193-kube-api-access-hwfgj\") pod \"payload-processing-e2e-ait-ad90c52d-d779bc747-ngwgg\" (UID: \"9118c67d-7bca-44ad-afc8-102cb132e193\") " pod="openshift-ingress/payload-processing-e2e-ait-ad90c52d-d779bc747-ngwgg"
Nov 01 05:41:05.823805 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:05.823779    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ad90c52d-7b64b785f-kn8k5"]
Nov 01 05:41:05.824321 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:05.824308    5689 cpu_manager.go:414] "containerMap: removing container" podUID="9a009de5-76be-4db1-adec-efb9246d4ddf" containerName="istio-proxy"
Nov 01 05:41:05.824373 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:05.824322    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a009de5-76be-4db1-adec-efb9246d4ddf" containerName="istio-proxy"
Nov 01 05:41:05.827721 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:05.827706    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ad90c52d-7b64b785f-kn8k5"
Nov 01 05:41:05.890310 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:05.890281    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ad90c52d-7b64b785f-kn8k5"]
Nov 01 05:41:05.893607 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:05.893579    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9118c67d-7bca-44ad-afc8-102cb132e193-plugins-config-volume\") pod \"payload-processing-e2e-ait-ad90c52d-d779bc747-ngwgg\" (UID: \"9118c67d-7bca-44ad-afc8-102cb132e193\") " pod="openshift-ingress/payload-processing-e2e-ait-ad90c52d-d779bc747-ngwgg"
Nov 01 05:41:05.893764 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:05.893630    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwfgj\" (UniqueName: \"kubernetes.io/projected/9118c67d-7bca-44ad-afc8-102cb132e193-kube-api-access-hwfgj\") pod \"payload-processing-e2e-ait-ad90c52d-d779bc747-ngwgg\" (UID: \"9118c67d-7bca-44ad-afc8-102cb132e193\") " pod="openshift-ingress/payload-processing-e2e-ait-ad90c52d-d779bc747-ngwgg"
Nov 01 05:41:05.894420 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:05.894393    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9118c67d-7bca-44ad-afc8-102cb132e193-plugins-config-volume\") pod \"payload-processing-e2e-ait-ad90c52d-d779bc747-ngwgg\" (UID: \"9118c67d-7bca-44ad-afc8-102cb132e193\") " pod="openshift-ingress/payload-processing-e2e-ait-ad90c52d-d779bc747-ngwgg"
Nov 01 05:41:05.958197 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:05.958115    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwfgj\" (UniqueName: \"kubernetes.io/projected/9118c67d-7bca-44ad-afc8-102cb132e193-kube-api-access-hwfgj\") pod \"payload-processing-e2e-ait-ad90c52d-d779bc747-ngwgg\" (UID: \"9118c67d-7bca-44ad-afc8-102cb132e193\") " pod="openshift-ingress/payload-processing-e2e-ait-ad90c52d-d779bc747-ngwgg"
Nov 01 05:41:05.995440 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:05.995415    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5664a5e-da43-4b3b-8d52-3789fca3d216-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ad90c52d-7b64b785f-kn8k5\" (UID: \"e5664a5e-da43-4b3b-8d52-3789fca3d216\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ad90c52d-7b64b785f-kn8k5"
Nov 01 05:41:05.995625 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:05.995603    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mh8g\" (UniqueName: \"kubernetes.io/projected/e5664a5e-da43-4b3b-8d52-3789fca3d216-kube-api-access-9mh8g\") pod \"payload-pre-processing-e2e-ait-ad90c52d-7b64b785f-kn8k5\" (UID: \"e5664a5e-da43-4b3b-8d52-3789fca3d216\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ad90c52d-7b64b785f-kn8k5"
Nov 01 05:41:06.037549 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:06.037513    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ad90c52d-d779bc747-ngwgg"
Nov 01 05:41:06.097070 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:06.097032    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mh8g\" (UniqueName: \"kubernetes.io/projected/e5664a5e-da43-4b3b-8d52-3789fca3d216-kube-api-access-9mh8g\") pod \"payload-pre-processing-e2e-ait-ad90c52d-7b64b785f-kn8k5\" (UID: \"e5664a5e-da43-4b3b-8d52-3789fca3d216\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ad90c52d-7b64b785f-kn8k5"
Nov 01 05:41:06.097205 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:06.097186    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5664a5e-da43-4b3b-8d52-3789fca3d216-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ad90c52d-7b64b785f-kn8k5\" (UID: \"e5664a5e-da43-4b3b-8d52-3789fca3d216\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ad90c52d-7b64b785f-kn8k5"
Nov 01 05:41:06.097885 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:06.097863    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5664a5e-da43-4b3b-8d52-3789fca3d216-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ad90c52d-7b64b785f-kn8k5\" (UID: \"e5664a5e-da43-4b3b-8d52-3789fca3d216\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ad90c52d-7b64b785f-kn8k5"
Nov 01 05:41:06.160090 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:06.160038    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mh8g\" (UniqueName: \"kubernetes.io/projected/e5664a5e-da43-4b3b-8d52-3789fca3d216-kube-api-access-9mh8g\") pod \"payload-pre-processing-e2e-ait-ad90c52d-7b64b785f-kn8k5\" (UID: \"e5664a5e-da43-4b3b-8d52-3789fca3d216\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ad90c52d-7b64b785f-kn8k5"
Nov 01 05:41:06.262278 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:06.260533    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:41:06.416191 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:06.416167    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ad90c52d-d779bc747-ngwgg"]
Nov 01 05:41:06.419292 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:41:06.419266    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9118c67d_7bca_44ad_afc8_102cb132e193.slice/crio-75946495d18ce41d664ad148307a11ea9f3278e931e2c10382e683e37a6d33f6 WatchSource:0}: Error finding container 75946495d18ce41d664ad148307a11ea9f3278e931e2c10382e683e37a6d33f6: Status 404 returned error can't find the container with id 75946495d18ce41d664ad148307a11ea9f3278e931e2c10382e683e37a6d33f6
Nov 01 05:41:06.440629 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:06.440607    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ad90c52d-7b64b785f-kn8k5"
Nov 01 05:41:06.441527 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:06.441503    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:41:06.824021 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:06.823950    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ad90c52d-7b64b785f-kn8k5"]
Nov 01 05:41:06.826334 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:41:06.826309    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5664a5e_da43_4b3b_8d52_3789fca3d216.slice/crio-69947a328c80cdb62b389545bef65c6018d7cebcd2b5718f4d8f8515963603be WatchSource:0}: Error finding container 69947a328c80cdb62b389545bef65c6018d7cebcd2b5718f4d8f8515963603be: Status 404 returned error can't find the container with id 69947a328c80cdb62b389545bef65c6018d7cebcd2b5718f4d8f8515963603be
Nov 01 05:41:07.279836 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:07.279804    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ad90c52d-d779bc747-ngwgg" event={"ID":"9118c67d-7bca-44ad-afc8-102cb132e193","Type":"ContainerStarted","Data":"eaa5877ddc6c2fa25107eee78638e1a38e37d62b3a04992006c8956d36f1e2fa"}
Nov 01 05:41:07.280324 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:07.279858    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ad90c52d-d779bc747-ngwgg" event={"ID":"9118c67d-7bca-44ad-afc8-102cb132e193","Type":"ContainerStarted","Data":"75946495d18ce41d664ad148307a11ea9f3278e931e2c10382e683e37a6d33f6"}
Nov 01 05:41:07.280324 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:07.279910    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-ad90c52d-d779bc747-ngwgg"
Nov 01 05:41:07.281563 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:07.281532    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ad90c52d-7b64b785f-kn8k5" event={"ID":"e5664a5e-da43-4b3b-8d52-3789fca3d216","Type":"ContainerStarted","Data":"b5b39842f4ee53af50d7851b04c957b0897bfd3b6d26f4dc1a4417ff6e11f7a0"}
Nov 01 05:41:07.281691 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:07.281571    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ad90c52d-7b64b785f-kn8k5" event={"ID":"e5664a5e-da43-4b3b-8d52-3789fca3d216","Type":"ContainerStarted","Data":"69947a328c80cdb62b389545bef65c6018d7cebcd2b5718f4d8f8515963603be"}
Nov 01 05:41:07.282008 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:07.281989    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ad90c52d-7b64b785f-kn8k5"
Nov 01 05:41:07.396710 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:07.396658    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-ad90c52d-d779bc747-ngwgg" podStartSLOduration=2.396646373 podStartE2EDuration="2.396646373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:41:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:41:07.394709166 +0000 UTC m=+2245.465505878" watchObservedRunningTime="2026-11-01 05:41:07.396646373 +0000 UTC m=+2245.467443093"
Nov 01 05:41:07.533849 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:07.531919    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-ad90c52d-7b64b785f-kn8k5" podStartSLOduration=2.531895157 podStartE2EDuration="2.531895157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:41:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:41:07.525354672 +0000 UTC m=+2245.596151392" watchObservedRunningTime="2026-11-01 05:41:07.531895157 +0000 UTC m=+2245.602691901"
Nov 01 05:41:11.030358 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:11.030308    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:41:15.818980 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:15.818944    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:41:18.290982 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:18.290951    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-ad90c52d-d779bc747-ngwgg"
Nov 01 05:41:19.114638 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:19.114605    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:41:19.299590 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:19.299195    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ad90c52d-7b64b785f-kn8k5"
Nov 01 05:41:19.299590 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:19.299367    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:41:20.004780 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:20.004748    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:41:20.050703 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:20.050611    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-577bl4r68"]
Nov 01 05:41:20.056884 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:20.056858    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-577bl4r68"
Nov 01 05:41:20.085945 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:20.085921    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-577bl4r68"]
Nov 01 05:41:20.132292 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:20.132259    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-dee6aa\"/\"agc-cred-2e04ffa38ad1\""
Nov 01 05:41:20.132292 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:20.132269    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-dee6aa\"/\"agc-cred-3bbeecafc00b\""
Nov 01 05:41:20.132504 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:20.132334    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-dee6aa\"/\"agc-cred-500efa28b76e\""
Nov 01 05:41:20.132504 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:20.132365    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-dee6aa\"/\"routing-overlay\""
Nov 01 05:41:20.132504 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:20.132391    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-dee6aa\"/\"payload-processing-external-model-e2e--19b5d07b-dockercfg-89znj\""
Nov 01 05:41:20.158422 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:20.158399    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h77pj\" (UniqueName: \"kubernetes.io/projected/2a7e7a74-357c-4c71-9d60-1f7384011e91-kube-api-access-h77pj\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-577bl4r68\" (UID: \"2a7e7a74-357c-4c71-9d60-1f7384011e91\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-577bl4r68"
Nov 01 05:41:20.158552 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:20.158435    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a7e7a74-357c-4c71-9d60-1f7384011e91-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-577bl4r68\" (UID: \"2a7e7a74-357c-4c71-9d60-1f7384011e91\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-577bl4r68"
Nov 01 05:41:20.158614 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:20.158599    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2a7e7a74-357c-4c71-9d60-1f7384011e91-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-577bl4r68\" (UID: \"2a7e7a74-357c-4c71-9d60-1f7384011e91\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-577bl4r68"
Nov 01 05:41:20.158689 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:20.158674    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2a7e7a74-357c-4c71-9d60-1f7384011e91-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-577bl4r68\" (UID: \"2a7e7a74-357c-4c71-9d60-1f7384011e91\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-577bl4r68"
Nov 01 05:41:20.159277 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:20.159262    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-dee6aa\"/\"kube-root-ca.crt\""
Nov 01 05:41:20.159349 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:20.159275    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-dee6aa\"/\"openshift-service-ca.crt\""
Nov 01 05:41:20.159349 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:20.159295    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-dee6aa\"/\"payload-processing-external-model-plugins-e2e-extmod-a-dee6aa\""
Nov 01 05:41:20.259583 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:20.259517    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2a7e7a74-357c-4c71-9d60-1f7384011e91-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-577bl4r68\" (UID: \"2a7e7a74-357c-4c71-9d60-1f7384011e91\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-577bl4r68"
Nov 01 05:41:20.259583 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:20.259566    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2a7e7a74-357c-4c71-9d60-1f7384011e91-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-577bl4r68\" (UID: \"2a7e7a74-357c-4c71-9d60-1f7384011e91\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-577bl4r68"
Nov 01 05:41:20.259720 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:20.259640    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h77pj\" (UniqueName: \"kubernetes.io/projected/2a7e7a74-357c-4c71-9d60-1f7384011e91-kube-api-access-h77pj\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-577bl4r68\" (UID: \"2a7e7a74-357c-4c71-9d60-1f7384011e91\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-577bl4r68"
Nov 01 05:41:20.259720 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:20.259676    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a7e7a74-357c-4c71-9d60-1f7384011e91-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-577bl4r68\" (UID: \"2a7e7a74-357c-4c71-9d60-1f7384011e91\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-577bl4r68"
Nov 01 05:41:20.260177 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:20.260159    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2a7e7a74-357c-4c71-9d60-1f7384011e91-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-577bl4r68\" (UID: \"2a7e7a74-357c-4c71-9d60-1f7384011e91\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-577bl4r68"
Nov 01 05:41:20.260396 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:20.260379    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a7e7a74-357c-4c71-9d60-1f7384011e91-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-577bl4r68\" (UID: \"2a7e7a74-357c-4c71-9d60-1f7384011e91\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-577bl4r68"
Nov 01 05:41:20.262286 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:20.262256    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2a7e7a74-357c-4c71-9d60-1f7384011e91-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-577bl4r68\" (UID: \"2a7e7a74-357c-4c71-9d60-1f7384011e91\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-577bl4r68"
Nov 01 05:41:20.320345 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:20.320318    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h77pj\" (UniqueName: \"kubernetes.io/projected/2a7e7a74-357c-4c71-9d60-1f7384011e91-kube-api-access-h77pj\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-577bl4r68\" (UID: \"2a7e7a74-357c-4c71-9d60-1f7384011e91\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-577bl4r68"
Nov 01 05:41:20.372032 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:20.372007    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-577bl4r68"
Nov 01 05:41:20.700247 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:20.700217    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:41:20.747776 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:20.747750    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-577bl4r68"]
Nov 01 05:41:20.794501 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:41:20.786736    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a7e7a74_357c_4c71_9d60_1f7384011e91.slice/crio-3cc89b29ccc1b8a3750aec0155c17874119b01c8aee1c53a0971345853ee9ef8 WatchSource:0}: Error finding container 3cc89b29ccc1b8a3750aec0155c17874119b01c8aee1c53a0971345853ee9ef8: Status 404 returned error can't find the container with id 3cc89b29ccc1b8a3750aec0155c17874119b01c8aee1c53a0971345853ee9ef8
Nov 01 05:41:21.362299 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:21.362267    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-577bl4r68" event={"ID":"2a7e7a74-357c-4c71-9d60-1f7384011e91","Type":"ContainerStarted","Data":"110b300f9288f73c2017eb9533182cc196775c8196bf2ae6ff8df6871bc70c64"}
Nov 01 05:41:21.362715 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:21.362315    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-577bl4r68" event={"ID":"2a7e7a74-357c-4c71-9d60-1f7384011e91","Type":"ContainerStarted","Data":"3cc89b29ccc1b8a3750aec0155c17874119b01c8aee1c53a0971345853ee9ef8"}
Nov 01 05:41:21.362715 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:21.362399    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-577bl4r68"
Nov 01 05:41:21.480390 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:21.480335    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-577bl4r68" podStartSLOduration=2.480322793 podStartE2EDuration="2.480322793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:41:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:41:21.478821273 +0000 UTC m=+2259.549617993" watchObservedRunningTime="2026-11-01 05:41:21.480322793 +0000 UTC m=+2259.551119514"
Nov 01 05:41:22.653744 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:22.653715    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-794cdbswp"]
Nov 01 05:41:22.657694 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:22.657674    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-794cdbswp"
Nov 01 05:41:22.702858 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:22.702827    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbc93b23-9a12-4f4c-951b-aa9e40e14d3e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-794cdbswp\" (UID: \"fbc93b23-9a12-4f4c-951b-aa9e40e14d3e\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-794cdbswp"
Nov 01 05:41:22.702858 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:22.702861    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6fkc\" (UniqueName: \"kubernetes.io/projected/fbc93b23-9a12-4f4c-951b-aa9e40e14d3e-kube-api-access-p6fkc\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-794cdbswp\" (UID: \"fbc93b23-9a12-4f4c-951b-aa9e40e14d3e\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-794cdbswp"
Nov 01 05:41:22.703027 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:22.702890    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fbc93b23-9a12-4f4c-951b-aa9e40e14d3e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-794cdbswp\" (UID: \"fbc93b23-9a12-4f4c-951b-aa9e40e14d3e\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-794cdbswp"
Nov 01 05:41:22.703072 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:22.703034    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fbc93b23-9a12-4f4c-951b-aa9e40e14d3e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-794cdbswp\" (UID: \"fbc93b23-9a12-4f4c-951b-aa9e40e14d3e\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-794cdbswp"
Nov 01 05:41:22.710002 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:22.709964    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-c6e99c\"/\"agc-cred-3e39090bc554\""
Nov 01 05:41:22.710149 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:22.710137    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-c6e99c\"/\"agc-cred-17c481ce95fa\""
Nov 01 05:41:22.710207 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:22.710147    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-c6e99c\"/\"agc-cred-8eb61444db22\""
Nov 01 05:41:22.710207 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:22.710154    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-c6e99c\"/\"payload-processing-external-model-e2e--8ea5e171-dockercfg-hr8xq\""
Nov 01 05:41:22.759686 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:22.759650    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c6e99c\"/\"kube-root-ca.crt\""
Nov 01 05:41:22.759686 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:22.759652    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c6e99c\"/\"openshift-service-ca.crt\""
Nov 01 05:41:22.759868 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:22.759652    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c6e99c\"/\"routing-overlay\""
Nov 01 05:41:22.759868 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:22.759703    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c6e99c\"/\"payload-processing-external-model-plugins-e2e-extmod-b-c6e99c\""
Nov 01 05:41:22.762134 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:22.762114    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-794cdbswp"]
Nov 01 05:41:22.803815 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:22.803784    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbc93b23-9a12-4f4c-951b-aa9e40e14d3e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-794cdbswp\" (UID: \"fbc93b23-9a12-4f4c-951b-aa9e40e14d3e\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-794cdbswp"
Nov 01 05:41:22.803941 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:22.803821    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6fkc\" (UniqueName: \"kubernetes.io/projected/fbc93b23-9a12-4f4c-951b-aa9e40e14d3e-kube-api-access-p6fkc\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-794cdbswp\" (UID: \"fbc93b23-9a12-4f4c-951b-aa9e40e14d3e\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-794cdbswp"
Nov 01 05:41:22.803941 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:22.803868    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fbc93b23-9a12-4f4c-951b-aa9e40e14d3e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-794cdbswp\" (UID: \"fbc93b23-9a12-4f4c-951b-aa9e40e14d3e\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-794cdbswp"
Nov 01 05:41:22.804053 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:22.803978    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fbc93b23-9a12-4f4c-951b-aa9e40e14d3e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-794cdbswp\" (UID: \"fbc93b23-9a12-4f4c-951b-aa9e40e14d3e\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-794cdbswp"
Nov 01 05:41:22.804546 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:22.804517    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fbc93b23-9a12-4f4c-951b-aa9e40e14d3e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-794cdbswp\" (UID: \"fbc93b23-9a12-4f4c-951b-aa9e40e14d3e\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-794cdbswp"
Nov 01 05:41:22.804658 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:22.804526    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbc93b23-9a12-4f4c-951b-aa9e40e14d3e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-794cdbswp\" (UID: \"fbc93b23-9a12-4f4c-951b-aa9e40e14d3e\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-794cdbswp"
Nov 01 05:41:22.806635 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:22.806619    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fbc93b23-9a12-4f4c-951b-aa9e40e14d3e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-794cdbswp\" (UID: \"fbc93b23-9a12-4f4c-951b-aa9e40e14d3e\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-794cdbswp"
Nov 01 05:41:22.864350 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:22.864320    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6fkc\" (UniqueName: \"kubernetes.io/projected/fbc93b23-9a12-4f4c-951b-aa9e40e14d3e-kube-api-access-p6fkc\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-794cdbswp\" (UID: \"fbc93b23-9a12-4f4c-951b-aa9e40e14d3e\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-794cdbswp"
Nov 01 05:41:22.971892 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:22.971828    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-794cdbswp"
Nov 01 05:41:23.351507 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:23.351482    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-794cdbswp"]
Nov 01 05:41:23.354229 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:41:23.354201    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbc93b23_9a12_4f4c_951b_aa9e40e14d3e.slice/crio-ad36eed5e399482059d91173cd7324f50482c253064d29f471b3dd1cd3581018 WatchSource:0}: Error finding container ad36eed5e399482059d91173cd7324f50482c253064d29f471b3dd1cd3581018: Status 404 returned error can't find the container with id ad36eed5e399482059d91173cd7324f50482c253064d29f471b3dd1cd3581018
Nov 01 05:41:23.376506 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:23.376447    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-794cdbswp" event={"ID":"fbc93b23-9a12-4f4c-951b-aa9e40e14d3e","Type":"ContainerStarted","Data":"ad36eed5e399482059d91173cd7324f50482c253064d29f471b3dd1cd3581018"}
Nov 01 05:41:24.383421 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:24.383389    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-794cdbswp" event={"ID":"fbc93b23-9a12-4f4c-951b-aa9e40e14d3e","Type":"ContainerStarted","Data":"e62918f6d2c9943087876a534880df49cd8fe722bc287a32a2d9439701ec1e47"}
Nov 01 05:41:24.383827 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:24.383461    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-794cdbswp"
Nov 01 05:41:24.502110 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:24.502062    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-794cdbswp" podStartSLOduration=2.502043868 podStartE2EDuration="2.502043868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:41:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:41:24.499644844 +0000 UTC m=+2262.570441582" watchObservedRunningTime="2026-11-01 05:41:24.502043868 +0000 UTC m=+2262.572840592"
Nov 01 05:41:25.045683 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:25.045648    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:41:25.228798 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:25.228758    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:41:25.450459 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:41:25.450416    5689 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-c6e99c/agc-cred-3e39090bc554: secret "agc-cred-3e39090bc554" not found
Nov 01 05:41:25.450857 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:41:25.450466    5689 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-794cdbswp: secret "agc-cred-3e39090bc554" not found
Nov 01 05:41:25.450857 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:41:25.450526    5689 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fbc93b23-9a12-4f4c-951b-aa9e40e14d3e-provider-credentials podName:fbc93b23-9a12-4f4c-951b-aa9e40e14d3e nodeName:}" failed. No retries permitted until 2026-11-01 05:41:25.950506878 +0000 UTC m=+2264.021303577 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/fbc93b23-9a12-4f4c-951b-aa9e40e14d3e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-c6e99c-794cdbswp" (UID: "fbc93b23-9a12-4f4c-951b-aa9e40e14d3e") : secret "agc-cred-3e39090bc554" not found
Nov 01 05:41:25.737810 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:25.737729    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:41:25.957221 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:41:25.957195    5689 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-c6e99c/agc-cred-3e39090bc554: secret "agc-cred-3e39090bc554" not found
Nov 01 05:41:25.957221 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:41:25.957219    5689 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-794cdbswp: secret "agc-cred-3e39090bc554" not found
Nov 01 05:41:25.957430 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:41:25.957266    5689 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fbc93b23-9a12-4f4c-951b-aa9e40e14d3e-provider-credentials podName:fbc93b23-9a12-4f4c-951b-aa9e40e14d3e nodeName:}" failed. No retries permitted until 2026-11-01 05:41:26.957253748 +0000 UTC m=+2265.028050460 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/fbc93b23-9a12-4f4c-951b-aa9e40e14d3e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-c6e99c-794cdbswp" (UID: "fbc93b23-9a12-4f4c-951b-aa9e40e14d3e") : secret "agc-cred-3e39090bc554" not found
Nov 01 05:41:26.966823 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:41:26.966786    5689 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-c6e99c/agc-cred-3e39090bc554: secret "agc-cred-3e39090bc554" not found
Nov 01 05:41:26.966823 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:41:26.966819    5689 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-794cdbswp: secret "agc-cred-3e39090bc554" not found
Nov 01 05:41:26.967249 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:41:26.966877    5689 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fbc93b23-9a12-4f4c-951b-aa9e40e14d3e-provider-credentials podName:fbc93b23-9a12-4f4c-951b-aa9e40e14d3e nodeName:}" failed. No retries permitted until 2026-11-01 05:41:28.9668631 +0000 UTC m=+2267.037659799 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/fbc93b23-9a12-4f4c-951b-aa9e40e14d3e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-c6e99c-794cdbswp" (UID: "fbc93b23-9a12-4f4c-951b-aa9e40e14d3e") : secret "agc-cred-3e39090bc554" not found
Nov 01 05:41:27.371364 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:27.371290    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:41:27.634419 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:27.634342    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-577bl4r68"]
Nov 01 05:41:27.637075 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:27.637046    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-577bl4r68" podUID="2a7e7a74-357c-4c71-9d60-1f7384011e91" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 05:41:27.668172 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:27.668146    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c675td6z"]
Nov 01 05:41:27.673308 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:27.673291    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c675td6z"
Nov 01 05:41:27.748084 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:27.748046    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c675td6z"]
Nov 01 05:41:27.832661 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:27.832626    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4daddd7-ac26-4180-8a37-c273bc6c9f6d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-5c675td6z\" (UID: \"c4daddd7-ac26-4180-8a37-c273bc6c9f6d\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c675td6z"
Nov 01 05:41:27.835265 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:27.835230    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c4daddd7-ac26-4180-8a37-c273bc6c9f6d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-5c675td6z\" (UID: \"c4daddd7-ac26-4180-8a37-c273bc6c9f6d\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c675td6z"
Nov 01 05:41:27.835393 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:27.835293    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-628td\" (UniqueName: \"kubernetes.io/projected/c4daddd7-ac26-4180-8a37-c273bc6c9f6d-kube-api-access-628td\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-5c675td6z\" (UID: \"c4daddd7-ac26-4180-8a37-c273bc6c9f6d\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c675td6z"
Nov 01 05:41:27.835393 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:27.835387    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c4daddd7-ac26-4180-8a37-c273bc6c9f6d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-5c675td6z\" (UID: \"c4daddd7-ac26-4180-8a37-c273bc6c9f6d\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c675td6z"
Nov 01 05:41:27.936411 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:27.936376    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c4daddd7-ac26-4180-8a37-c273bc6c9f6d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-5c675td6z\" (UID: \"c4daddd7-ac26-4180-8a37-c273bc6c9f6d\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c675td6z"
Nov 01 05:41:27.936621 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:27.936511    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4daddd7-ac26-4180-8a37-c273bc6c9f6d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-5c675td6z\" (UID: \"c4daddd7-ac26-4180-8a37-c273bc6c9f6d\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c675td6z"
Nov 01 05:41:27.936621 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:27.936560    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c4daddd7-ac26-4180-8a37-c273bc6c9f6d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-5c675td6z\" (UID: \"c4daddd7-ac26-4180-8a37-c273bc6c9f6d\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c675td6z"
Nov 01 05:41:27.936621 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:27.936585    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-628td\" (UniqueName: \"kubernetes.io/projected/c4daddd7-ac26-4180-8a37-c273bc6c9f6d-kube-api-access-628td\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-5c675td6z\" (UID: \"c4daddd7-ac26-4180-8a37-c273bc6c9f6d\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c675td6z"
Nov 01 05:41:27.937265 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:27.937240    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c4daddd7-ac26-4180-8a37-c273bc6c9f6d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-5c675td6z\" (UID: \"c4daddd7-ac26-4180-8a37-c273bc6c9f6d\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c675td6z"
Nov 01 05:41:27.937368 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:27.937276    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4daddd7-ac26-4180-8a37-c273bc6c9f6d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-5c675td6z\" (UID: \"c4daddd7-ac26-4180-8a37-c273bc6c9f6d\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c675td6z"
Nov 01 05:41:27.939009 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:27.938988    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c4daddd7-ac26-4180-8a37-c273bc6c9f6d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-5c675td6z\" (UID: \"c4daddd7-ac26-4180-8a37-c273bc6c9f6d\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c675td6z"
Nov 01 05:41:27.999446 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:27.999419    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-628td\" (UniqueName: \"kubernetes.io/projected/c4daddd7-ac26-4180-8a37-c273bc6c9f6d-kube-api-access-628td\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-5c675td6z\" (UID: \"c4daddd7-ac26-4180-8a37-c273bc6c9f6d\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c675td6z"
Nov 01 05:41:28.290216 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:28.290143    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c675td6z"
Nov 01 05:41:28.668309 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:28.668265    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c675td6z"]
Nov 01 05:41:28.671026 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:41:28.671001    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4daddd7_ac26_4180_8a37_c273bc6c9f6d.slice/crio-2656452b49ed4562bc9415982c5c024e40e76a0686ae991ac605db6a1164bcab WatchSource:0}: Error finding container 2656452b49ed4562bc9415982c5c024e40e76a0686ae991ac605db6a1164bcab: Status 404 returned error can't find the container with id 2656452b49ed4562bc9415982c5c024e40e76a0686ae991ac605db6a1164bcab
Nov 01 05:41:29.052735 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:41:29.052657    5689 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-c6e99c/agc-cred-3e39090bc554: secret "agc-cred-3e39090bc554" not found
Nov 01 05:41:29.052735 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:41:29.052685    5689 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-794cdbswp: secret "agc-cred-3e39090bc554" not found
Nov 01 05:41:29.053131 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:41:29.052742    5689 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fbc93b23-9a12-4f4c-951b-aa9e40e14d3e-provider-credentials podName:fbc93b23-9a12-4f4c-951b-aa9e40e14d3e nodeName:}" failed. No retries permitted until 2026-11-01 05:41:33.052725826 +0000 UTC m=+2271.123522540 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/fbc93b23-9a12-4f4c-951b-aa9e40e14d3e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-c6e99c-794cdbswp" (UID: "fbc93b23-9a12-4f4c-951b-aa9e40e14d3e") : secret "agc-cred-3e39090bc554" not found
Nov 01 05:41:29.413271 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:29.413238    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c675td6z" event={"ID":"c4daddd7-ac26-4180-8a37-c273bc6c9f6d","Type":"ContainerStarted","Data":"83d4c2eb5ed5c784169e12c10346b524c79f37673c50c9e22f5663eec47a5819"}
Nov 01 05:41:29.413434 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:29.413319    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c675td6z" event={"ID":"c4daddd7-ac26-4180-8a37-c273bc6c9f6d","Type":"ContainerStarted","Data":"2656452b49ed4562bc9415982c5c024e40e76a0686ae991ac605db6a1164bcab"}
Nov 01 05:41:29.413434 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:29.413340    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c675td6z"
Nov 01 05:41:29.533948 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:29.533903    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c675td6z" podStartSLOduration=2.533889742 podStartE2EDuration="2.533889742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:41:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:41:29.532549074 +0000 UTC m=+2267.603345795" watchObservedRunningTime="2026-11-01 05:41:29.533889742 +0000 UTC m=+2267.604686462"
Nov 01 05:41:30.070249 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:41:30.070217    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:41:30.247468 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:30.247423    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-794cdbswp"]
Nov 01 05:41:30.250039 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:30.250008    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-794cdbswp" podUID="fbc93b23-9a12-4f4c-951b-aa9e40e14d3e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 05:41:30.522930 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:30.522888    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:41:32.635654 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:32.635616    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-577bl4r68" podUID="2a7e7a74-357c-4c71-9d60-1f7384011e91" containerName="payload-processing" containerID="cri-o://110b300f9288f73c2017eb9533182cc196775c8196bf2ae6ff8df6871bc70c64" gracePeriod=25
Nov 01 05:41:32.889701 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:32.889640    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-577bl4r68"
Nov 01 05:41:32.994348 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:32.994322    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a7e7a74-357c-4c71-9d60-1f7384011e91-kube-api-access-h77pj\" (UniqueName: \"kubernetes.io/projected/2a7e7a74-357c-4c71-9d60-1f7384011e91-kube-api-access-h77pj\") pod \"2a7e7a74-357c-4c71-9d60-1f7384011e91\" (UID: \"2a7e7a74-357c-4c71-9d60-1f7384011e91\") "
Nov 01 05:41:32.994532 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:32.994440    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a7e7a74-357c-4c71-9d60-1f7384011e91-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a7e7a74-357c-4c71-9d60-1f7384011e91-plugins-config-volume\") pod \"2a7e7a74-357c-4c71-9d60-1f7384011e91\" (UID: \"2a7e7a74-357c-4c71-9d60-1f7384011e91\") "
Nov 01 05:41:32.994605 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:32.994587    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a7e7a74-357c-4c71-9d60-1f7384011e91-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2a7e7a74-357c-4c71-9d60-1f7384011e91-provider-credentials\") pod \"2a7e7a74-357c-4c71-9d60-1f7384011e91\" (UID: \"2a7e7a74-357c-4c71-9d60-1f7384011e91\") "
Nov 01 05:41:32.994724 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:32.994707    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a7e7a74-357c-4c71-9d60-1f7384011e91-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2a7e7a74-357c-4c71-9d60-1f7384011e91-routing-overlay-volume\") pod \"2a7e7a74-357c-4c71-9d60-1f7384011e91\" (UID: \"2a7e7a74-357c-4c71-9d60-1f7384011e91\") "
Nov 01 05:41:32.994835 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:32.994811    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a7e7a74-357c-4c71-9d60-1f7384011e91-plugins-config-volume" pod "2a7e7a74-357c-4c71-9d60-1f7384011e91" (UID: "2a7e7a74-357c-4c71-9d60-1f7384011e91"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:41:32.995040 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:32.995017    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a7e7a74-357c-4c71-9d60-1f7384011e91-routing-overlay-volume" pod "2a7e7a74-357c-4c71-9d60-1f7384011e91" (UID: "2a7e7a74-357c-4c71-9d60-1f7384011e91"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:41:32.995367 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:32.995319    5689 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2a7e7a74-357c-4c71-9d60-1f7384011e91-routing-overlay-volume\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:41:32.995367 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:32.995345    5689 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a7e7a74-357c-4c71-9d60-1f7384011e91-plugins-config-volume\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:41:32.996651 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:32.996627    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a7e7a74-357c-4c71-9d60-1f7384011e91-kube-api-access-h77pj" pod "2a7e7a74-357c-4c71-9d60-1f7384011e91" (UID: "2a7e7a74-357c-4c71-9d60-1f7384011e91"). InnerVolumeSpecName "kube-api-access-h77pj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:41:32.996736 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:32.996679    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a7e7a74-357c-4c71-9d60-1f7384011e91-provider-credentials" pod "2a7e7a74-357c-4c71-9d60-1f7384011e91" (UID: "2a7e7a74-357c-4c71-9d60-1f7384011e91"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:41:33.096313 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:33.096276    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h77pj\" (UniqueName: \"kubernetes.io/projected/2a7e7a74-357c-4c71-9d60-1f7384011e91-kube-api-access-h77pj\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:41:33.096313 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:33.096304    5689 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2a7e7a74-357c-4c71-9d60-1f7384011e91-provider-credentials\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:41:33.096313 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:41:33.096316    5689 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-c6e99c/agc-cred-3e39090bc554: secret "agc-cred-3e39090bc554" not found
Nov 01 05:41:33.096605 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:41:33.096335    5689 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-794cdbswp: secret "agc-cred-3e39090bc554" not found
Nov 01 05:41:33.096605 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:41:33.096393    5689 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fbc93b23-9a12-4f4c-951b-aa9e40e14d3e-provider-credentials podName:fbc93b23-9a12-4f4c-951b-aa9e40e14d3e nodeName:}" failed. No retries permitted until 2026-11-01 05:41:41.096377443 +0000 UTC m=+2279.167174161 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/fbc93b23-9a12-4f4c-951b-aa9e40e14d3e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-c6e99c-794cdbswp" (UID: "fbc93b23-9a12-4f4c-951b-aa9e40e14d3e") : secret "agc-cred-3e39090bc554" not found
Nov 01 05:41:33.436009 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:33.435968    5689 generic.go:356] "Generic (PLEG): container finished" podID="2a7e7a74-357c-4c71-9d60-1f7384011e91" containerID="110b300f9288f73c2017eb9533182cc196775c8196bf2ae6ff8df6871bc70c64" exitCode=0
Nov 01 05:41:33.436178 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:33.436015    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-577bl4r68"
Nov 01 05:41:33.436178 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:33.436048    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-577bl4r68" event={"ID":"2a7e7a74-357c-4c71-9d60-1f7384011e91","Type":"ContainerDied","Data":"110b300f9288f73c2017eb9533182cc196775c8196bf2ae6ff8df6871bc70c64"}
Nov 01 05:41:33.436178 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:33.436100    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-577bl4r68" event={"ID":"2a7e7a74-357c-4c71-9d60-1f7384011e91","Type":"ContainerDied","Data":"3cc89b29ccc1b8a3750aec0155c17874119b01c8aee1c53a0971345853ee9ef8"}
Nov 01 05:41:33.436178 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:33.436121    5689 scope.go:122] "RemoveContainer" containerID="110b300f9288f73c2017eb9533182cc196775c8196bf2ae6ff8df6871bc70c64"
Nov 01 05:41:33.446351 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:33.446335    5689 scope.go:122] "RemoveContainer" containerID="110b300f9288f73c2017eb9533182cc196775c8196bf2ae6ff8df6871bc70c64"
Nov 01 05:41:33.446638 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:41:33.446618    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"110b300f9288f73c2017eb9533182cc196775c8196bf2ae6ff8df6871bc70c64\": container with ID starting with 110b300f9288f73c2017eb9533182cc196775c8196bf2ae6ff8df6871bc70c64 not found: ID does not exist" containerID="110b300f9288f73c2017eb9533182cc196775c8196bf2ae6ff8df6871bc70c64"
Nov 01 05:41:33.446710 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:33.446646    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"110b300f9288f73c2017eb9533182cc196775c8196bf2ae6ff8df6871bc70c64"} err="failed to get container status \"110b300f9288f73c2017eb9533182cc196775c8196bf2ae6ff8df6871bc70c64\": rpc error: code = NotFound desc = could not find container \"110b300f9288f73c2017eb9533182cc196775c8196bf2ae6ff8df6871bc70c64\": container with ID starting with 110b300f9288f73c2017eb9533182cc196775c8196bf2ae6ff8df6871bc70c64 not found: ID does not exist"
Nov 01 05:41:33.599808 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:33.599776    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-577bl4r68"]
Nov 01 05:41:33.603994 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:33.603966    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-577bl4r68"]
Nov 01 05:41:34.068847 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:34.068815    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a7e7a74-357c-4c71-9d60-1f7384011e91" path="/var/lib/kubelet/pods/2a7e7a74-357c-4c71-9d60-1f7384011e91/volumes"
Nov 01 05:41:35.248723 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:35.248681    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-794cdbswp" podUID="fbc93b23-9a12-4f4c-951b-aa9e40e14d3e" containerName="payload-processing" containerID="cri-o://e62918f6d2c9943087876a534880df49cd8fe722bc287a32a2d9439701ec1e47" gracePeriod=25
Nov 01 05:41:35.257229 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:35.257199    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:41:35.490854 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:35.490814    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:41:35.510753 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:35.510666    5689 generic.go:356] "Generic (PLEG): container finished" podID="fbc93b23-9a12-4f4c-951b-aa9e40e14d3e" containerID="e62918f6d2c9943087876a534880df49cd8fe722bc287a32a2d9439701ec1e47" exitCode=0
Nov 01 05:41:35.510920 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:35.510743    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-794cdbswp" event={"ID":"fbc93b23-9a12-4f4c-951b-aa9e40e14d3e","Type":"ContainerDied","Data":"e62918f6d2c9943087876a534880df49cd8fe722bc287a32a2d9439701ec1e47"}
Nov 01 05:41:35.548442 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:35.548422    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-794cdbswp"
Nov 01 05:41:35.621926 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:35.621894    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbc93b23-9a12-4f4c-951b-aa9e40e14d3e-kube-api-access-p6fkc\" (UniqueName: \"kubernetes.io/projected/fbc93b23-9a12-4f4c-951b-aa9e40e14d3e-kube-api-access-p6fkc\") pod \"fbc93b23-9a12-4f4c-951b-aa9e40e14d3e\" (UID: \"fbc93b23-9a12-4f4c-951b-aa9e40e14d3e\") "
Nov 01 05:41:35.622056 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:35.621944    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbc93b23-9a12-4f4c-951b-aa9e40e14d3e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fbc93b23-9a12-4f4c-951b-aa9e40e14d3e-routing-overlay-volume\") pod \"fbc93b23-9a12-4f4c-951b-aa9e40e14d3e\" (UID: \"fbc93b23-9a12-4f4c-951b-aa9e40e14d3e\") "
Nov 01 05:41:35.622056 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:35.622000    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbc93b23-9a12-4f4c-951b-aa9e40e14d3e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/fbc93b23-9a12-4f4c-951b-aa9e40e14d3e-provider-credentials\") pod \"fbc93b23-9a12-4f4c-951b-aa9e40e14d3e\" (UID: \"fbc93b23-9a12-4f4c-951b-aa9e40e14d3e\") "
Nov 01 05:41:35.622146 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:35.622096    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbc93b23-9a12-4f4c-951b-aa9e40e14d3e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbc93b23-9a12-4f4c-951b-aa9e40e14d3e-plugins-config-volume\") pod \"fbc93b23-9a12-4f4c-951b-aa9e40e14d3e\" (UID: \"fbc93b23-9a12-4f4c-951b-aa9e40e14d3e\") "
Nov 01 05:41:35.622422 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:35.622333    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbc93b23-9a12-4f4c-951b-aa9e40e14d3e-routing-overlay-volume" pod "fbc93b23-9a12-4f4c-951b-aa9e40e14d3e" (UID: "fbc93b23-9a12-4f4c-951b-aa9e40e14d3e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:41:35.622586 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:35.622559    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbc93b23-9a12-4f4c-951b-aa9e40e14d3e-plugins-config-volume" pod "fbc93b23-9a12-4f4c-951b-aa9e40e14d3e" (UID: "fbc93b23-9a12-4f4c-951b-aa9e40e14d3e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:41:35.622662 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:35.622643    5689 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fbc93b23-9a12-4f4c-951b-aa9e40e14d3e-routing-overlay-volume\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:41:35.624216 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:35.624198    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbc93b23-9a12-4f4c-951b-aa9e40e14d3e-kube-api-access-p6fkc" pod "fbc93b23-9a12-4f4c-951b-aa9e40e14d3e" (UID: "fbc93b23-9a12-4f4c-951b-aa9e40e14d3e"). InnerVolumeSpecName "kube-api-access-p6fkc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:41:35.624273 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:35.624250    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbc93b23-9a12-4f4c-951b-aa9e40e14d3e-provider-credentials" pod "fbc93b23-9a12-4f4c-951b-aa9e40e14d3e" (UID: "fbc93b23-9a12-4f4c-951b-aa9e40e14d3e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:41:35.691020 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:35.690986    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:41:35.723536 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:35.723508    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6fkc\" (UniqueName: \"kubernetes.io/projected/fbc93b23-9a12-4f4c-951b-aa9e40e14d3e-kube-api-access-p6fkc\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:41:35.723536 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:35.723533    5689 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fbc93b23-9a12-4f4c-951b-aa9e40e14d3e-provider-credentials\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:41:35.723536 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:35.723543    5689 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbc93b23-9a12-4f4c-951b-aa9e40e14d3e-plugins-config-volume\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:41:36.518268 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:36.518237    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-794cdbswp" event={"ID":"fbc93b23-9a12-4f4c-951b-aa9e40e14d3e","Type":"ContainerDied","Data":"ad36eed5e399482059d91173cd7324f50482c253064d29f471b3dd1cd3581018"}
Nov 01 05:41:36.518268 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:36.518259    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-794cdbswp"
Nov 01 05:41:36.518840 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:36.518291    5689 scope.go:122] "RemoveContainer" containerID="e62918f6d2c9943087876a534880df49cd8fe722bc287a32a2d9439701ec1e47"
Nov 01 05:41:36.689213 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:36.689174    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-794cdbswp"]
Nov 01 05:41:36.738977 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:36.738948    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-794cdbswp"]
Nov 01 05:41:38.076008 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:38.075967    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbc93b23-9a12-4f4c-951b-aa9e40e14d3e" path="/var/lib/kubelet/pods/fbc93b23-9a12-4f4c-951b-aa9e40e14d3e/volumes"
Nov 01 05:41:40.265415 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:40.265344    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:41:40.421406 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:40.421371    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c675td6z" podUID="c4daddd7-ac26-4180-8a37-c273bc6c9f6d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 05:41:45.048301 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:45.043632    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:41:49.845762 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:49.845705    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:41:50.248665 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:50.248635    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:41:50.430199 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:50.430161    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c675td6z" podUID="c4daddd7-ac26-4180-8a37-c273bc6c9f6d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 05:41:50.437844 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:50.437797    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:41:51.160018 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:51.159984    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c675td6z"]
Nov 01 05:41:51.161271 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:51.161242    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c675td6z" podUID="c4daddd7-ac26-4180-8a37-c273bc6c9f6d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 05:41:53.866375 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:53.866342    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-d989v9bzg"]
Nov 01 05:41:53.867064 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:53.867043    5689 cpu_manager.go:414] "containerMap: removing container" podUID="2a7e7a74-357c-4c71-9d60-1f7384011e91" containerName="payload-processing"
Nov 01 05:41:53.867064 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:53.867063    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a7e7a74-357c-4c71-9d60-1f7384011e91" containerName="payload-processing"
Nov 01 05:41:53.867208 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:53.867076    5689 cpu_manager.go:414] "containerMap: removing container" podUID="fbc93b23-9a12-4f4c-951b-aa9e40e14d3e" containerName="payload-processing"
Nov 01 05:41:53.867208 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:53.867082    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbc93b23-9a12-4f4c-951b-aa9e40e14d3e" containerName="payload-processing"
Nov 01 05:41:53.867208 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:53.867181    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a7e7a74-357c-4c71-9d60-1f7384011e91" containerName="payload-processing"
Nov 01 05:41:53.867208 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:53.867191    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbc93b23-9a12-4f4c-951b-aa9e40e14d3e" containerName="payload-processing"
Nov 01 05:41:53.872109 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:53.872093    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-d989v9bzg"
Nov 01 05:41:53.965314 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:53.965279    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/98140dfe-532f-416f-8f3b-b549977e7cc1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-d989v9bzg\" (UID: \"98140dfe-532f-416f-8f3b-b549977e7cc1\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-d989v9bzg"
Nov 01 05:41:53.965525 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:53.965468    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/98140dfe-532f-416f-8f3b-b549977e7cc1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-d989v9bzg\" (UID: \"98140dfe-532f-416f-8f3b-b549977e7cc1\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-d989v9bzg"
Nov 01 05:41:53.965610 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:53.965525    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98140dfe-532f-416f-8f3b-b549977e7cc1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-d989v9bzg\" (UID: \"98140dfe-532f-416f-8f3b-b549977e7cc1\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-d989v9bzg"
Nov 01 05:41:53.965610 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:53.965601    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29k7d\" (UniqueName: \"kubernetes.io/projected/98140dfe-532f-416f-8f3b-b549977e7cc1-kube-api-access-29k7d\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-d989v9bzg\" (UID: \"98140dfe-532f-416f-8f3b-b549977e7cc1\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-d989v9bzg"
Nov 01 05:41:53.973759 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:53.973741    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c6e99c\"/\"openshift-service-ca.crt\""
Nov 01 05:41:53.973759 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:53.973741    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c6e99c\"/\"payload-processing-external-model-plugins-e2e-extmod-b-c6e99c\""
Nov 01 05:41:53.973957 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:53.973860    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c6e99c\"/\"routing-overlay\""
Nov 01 05:41:53.973957 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:53.973860    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-c6e99c\"/\"agc-cred-17c481ce95fa\""
Nov 01 05:41:53.973957 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:53.973878    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-c6e99c\"/\"payload-processing-external-model-e2e--8ea5e171-dockercfg-hr8xq\""
Nov 01 05:41:53.974063 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:53.973976    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c6e99c\"/\"kube-root-ca.crt\""
Nov 01 05:41:53.976206 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:53.976187    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-d989v9bzg"]
Nov 01 05:41:54.066498 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:54.066440    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29k7d\" (UniqueName: \"kubernetes.io/projected/98140dfe-532f-416f-8f3b-b549977e7cc1-kube-api-access-29k7d\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-d989v9bzg\" (UID: \"98140dfe-532f-416f-8f3b-b549977e7cc1\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-d989v9bzg"
Nov 01 05:41:54.066649 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:54.066513    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/98140dfe-532f-416f-8f3b-b549977e7cc1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-d989v9bzg\" (UID: \"98140dfe-532f-416f-8f3b-b549977e7cc1\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-d989v9bzg"
Nov 01 05:41:54.066649 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:54.066612    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/98140dfe-532f-416f-8f3b-b549977e7cc1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-d989v9bzg\" (UID: \"98140dfe-532f-416f-8f3b-b549977e7cc1\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-d989v9bzg"
Nov 01 05:41:54.066649 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:54.066643    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98140dfe-532f-416f-8f3b-b549977e7cc1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-d989v9bzg\" (UID: \"98140dfe-532f-416f-8f3b-b549977e7cc1\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-d989v9bzg"
Nov 01 05:41:54.067335 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:54.067269    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/98140dfe-532f-416f-8f3b-b549977e7cc1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-d989v9bzg\" (UID: \"98140dfe-532f-416f-8f3b-b549977e7cc1\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-d989v9bzg"
Nov 01 05:41:54.067335 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:54.067372    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98140dfe-532f-416f-8f3b-b549977e7cc1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-d989v9bzg\" (UID: \"98140dfe-532f-416f-8f3b-b549977e7cc1\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-d989v9bzg"
Nov 01 05:41:54.069157 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:54.069137    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/98140dfe-532f-416f-8f3b-b549977e7cc1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-d989v9bzg\" (UID: \"98140dfe-532f-416f-8f3b-b549977e7cc1\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-d989v9bzg"
Nov 01 05:41:54.127387 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:54.127321    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29k7d\" (UniqueName: \"kubernetes.io/projected/98140dfe-532f-416f-8f3b-b549977e7cc1-kube-api-access-29k7d\") pod \"payload-processing-external-model-e2e-extmod-b-c6e99c-d989v9bzg\" (UID: \"98140dfe-532f-416f-8f3b-b549977e7cc1\") " pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-d989v9bzg"
Nov 01 05:41:54.185192 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:54.185166    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-d989v9bzg"
Nov 01 05:41:54.564223 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:54.564188    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-d989v9bzg"]
Nov 01 05:41:54.567709 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:41:54.567680    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98140dfe_532f_416f_8f3b_b549977e7cc1.slice/crio-40f18c24edebfce9f13a15bb561a80050fb636efc06303ff778a243a0c5a4eed WatchSource:0}: Error finding container 40f18c24edebfce9f13a15bb561a80050fb636efc06303ff778a243a0c5a4eed: Status 404 returned error can't find the container with id 40f18c24edebfce9f13a15bb561a80050fb636efc06303ff778a243a0c5a4eed
Nov 01 05:41:54.633987 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:54.633960    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-d989v9bzg" event={"ID":"98140dfe-532f-416f-8f3b-b549977e7cc1","Type":"ContainerStarted","Data":"f3fb80117d24ffeedd4aff63c341c1a6765d8b30cd5a74dcba2b112935633954"}
Nov 01 05:41:54.634097 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:54.634014    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-d989v9bzg" event={"ID":"98140dfe-532f-416f-8f3b-b549977e7cc1","Type":"ContainerStarted","Data":"40f18c24edebfce9f13a15bb561a80050fb636efc06303ff778a243a0c5a4eed"}
Nov 01 05:41:54.634097 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:54.634037    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-d989v9bzg"
Nov 01 05:41:54.681272 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:54.678754    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:41:54.751530 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:54.751473    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-d989v9bzg" podStartSLOduration=1.75144829 podStartE2EDuration="1.75144829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:41:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:41:54.75071269 +0000 UTC m=+2292.821509411" watchObservedRunningTime="2026-11-01 05:41:54.75144829 +0000 UTC m=+2292.822245032"
Nov 01 05:41:56.160592 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:56.160550    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c675td6z" podUID="c4daddd7-ac26-4180-8a37-c273bc6c9f6d" containerName="payload-processing" containerID="cri-o://83d4c2eb5ed5c784169e12c10346b524c79f37673c50c9e22f5663eec47a5819" gracePeriod=25
Nov 01 05:41:56.438043 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:56.438014    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c675td6z"
Nov 01 05:41:56.594182 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:56.594144    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4daddd7-ac26-4180-8a37-c273bc6c9f6d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c4daddd7-ac26-4180-8a37-c273bc6c9f6d-routing-overlay-volume\") pod \"c4daddd7-ac26-4180-8a37-c273bc6c9f6d\" (UID: \"c4daddd7-ac26-4180-8a37-c273bc6c9f6d\") "
Nov 01 05:41:56.594363 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:56.594186    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4daddd7-ac26-4180-8a37-c273bc6c9f6d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4daddd7-ac26-4180-8a37-c273bc6c9f6d-plugins-config-volume\") pod \"c4daddd7-ac26-4180-8a37-c273bc6c9f6d\" (UID: \"c4daddd7-ac26-4180-8a37-c273bc6c9f6d\") "
Nov 01 05:41:56.594363 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:56.594334    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4daddd7-ac26-4180-8a37-c273bc6c9f6d-kube-api-access-628td\" (UniqueName: \"kubernetes.io/projected/c4daddd7-ac26-4180-8a37-c273bc6c9f6d-kube-api-access-628td\") pod \"c4daddd7-ac26-4180-8a37-c273bc6c9f6d\" (UID: \"c4daddd7-ac26-4180-8a37-c273bc6c9f6d\") "
Nov 01 05:41:56.594551 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:56.594535    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4daddd7-ac26-4180-8a37-c273bc6c9f6d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c4daddd7-ac26-4180-8a37-c273bc6c9f6d-provider-credentials\") pod \"c4daddd7-ac26-4180-8a37-c273bc6c9f6d\" (UID: \"c4daddd7-ac26-4180-8a37-c273bc6c9f6d\") "
Nov 01 05:41:56.594663 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:56.594635    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4daddd7-ac26-4180-8a37-c273bc6c9f6d-routing-overlay-volume" pod "c4daddd7-ac26-4180-8a37-c273bc6c9f6d" (UID: "c4daddd7-ac26-4180-8a37-c273bc6c9f6d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:41:56.594775 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:56.594658    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4daddd7-ac26-4180-8a37-c273bc6c9f6d-plugins-config-volume" pod "c4daddd7-ac26-4180-8a37-c273bc6c9f6d" (UID: "c4daddd7-ac26-4180-8a37-c273bc6c9f6d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:41:56.594973 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:56.594956    5689 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c4daddd7-ac26-4180-8a37-c273bc6c9f6d-routing-overlay-volume\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:41:56.595033 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:56.594981    5689 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4daddd7-ac26-4180-8a37-c273bc6c9f6d-plugins-config-volume\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:41:56.596695 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:56.596671    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4daddd7-ac26-4180-8a37-c273bc6c9f6d-kube-api-access-628td" pod "c4daddd7-ac26-4180-8a37-c273bc6c9f6d" (UID: "c4daddd7-ac26-4180-8a37-c273bc6c9f6d"). InnerVolumeSpecName "kube-api-access-628td". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:41:56.596813 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:56.596798    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4daddd7-ac26-4180-8a37-c273bc6c9f6d-provider-credentials" pod "c4daddd7-ac26-4180-8a37-c273bc6c9f6d" (UID: "c4daddd7-ac26-4180-8a37-c273bc6c9f6d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:41:56.646126 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:56.646093    5689 generic.go:356] "Generic (PLEG): container finished" podID="c4daddd7-ac26-4180-8a37-c273bc6c9f6d" containerID="83d4c2eb5ed5c784169e12c10346b524c79f37673c50c9e22f5663eec47a5819" exitCode=0
Nov 01 05:41:56.646275 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:56.646141    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c675td6z"
Nov 01 05:41:56.646275 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:56.646169    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c675td6z" event={"ID":"c4daddd7-ac26-4180-8a37-c273bc6c9f6d","Type":"ContainerDied","Data":"83d4c2eb5ed5c784169e12c10346b524c79f37673c50c9e22f5663eec47a5819"}
Nov 01 05:41:56.646275 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:56.646221    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c675td6z" event={"ID":"c4daddd7-ac26-4180-8a37-c273bc6c9f6d","Type":"ContainerDied","Data":"2656452b49ed4562bc9415982c5c024e40e76a0686ae991ac605db6a1164bcab"}
Nov 01 05:41:56.646275 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:56.646248    5689 scope.go:122] "RemoveContainer" containerID="83d4c2eb5ed5c784169e12c10346b524c79f37673c50c9e22f5663eec47a5819"
Nov 01 05:41:56.658429 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:56.658413    5689 scope.go:122] "RemoveContainer" containerID="83d4c2eb5ed5c784169e12c10346b524c79f37673c50c9e22f5663eec47a5819"
Nov 01 05:41:56.658749 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:41:56.658725    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83d4c2eb5ed5c784169e12c10346b524c79f37673c50c9e22f5663eec47a5819\": container with ID starting with 83d4c2eb5ed5c784169e12c10346b524c79f37673c50c9e22f5663eec47a5819 not found: ID does not exist" containerID="83d4c2eb5ed5c784169e12c10346b524c79f37673c50c9e22f5663eec47a5819"
Nov 01 05:41:56.658805 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:56.658752    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83d4c2eb5ed5c784169e12c10346b524c79f37673c50c9e22f5663eec47a5819"} err="failed to get container status \"83d4c2eb5ed5c784169e12c10346b524c79f37673c50c9e22f5663eec47a5819\": rpc error: code = NotFound desc = could not find container \"83d4c2eb5ed5c784169e12c10346b524c79f37673c50c9e22f5663eec47a5819\": container with ID starting with 83d4c2eb5ed5c784169e12c10346b524c79f37673c50c9e22f5663eec47a5819 not found: ID does not exist"
Nov 01 05:41:56.696612 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:56.696554    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-628td\" (UniqueName: \"kubernetes.io/projected/c4daddd7-ac26-4180-8a37-c273bc6c9f6d-kube-api-access-628td\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:41:56.696612 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:56.696577    5689 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c4daddd7-ac26-4180-8a37-c273bc6c9f6d-provider-credentials\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:41:56.810034 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:56.810002    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c675td6z"]
Nov 01 05:41:56.813982 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:56.813959    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-5c675td6z"]
Nov 01 05:41:58.068288 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:58.068261    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4daddd7-ac26-4180-8a37-c273bc6c9f6d" path="/var/lib/kubelet/pods/c4daddd7-ac26-4180-8a37-c273bc6c9f6d/volumes"
Nov 01 05:41:59.307948 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:59.307914    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:41:59.486582 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:59.486553    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:41:59.668387 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:41:59.668357    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:42:00.069388 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:42:00.069313    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:42:04.309632 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:04.309602    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:42:05.642727 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:05.642681    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-d989v9bzg" podUID="98140dfe-532f-416f-8f3b-b549977e7cc1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 05:42:06.718890 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:06.718858    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:42:09.232850 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:09.232816    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:42:13.898690 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:13.898656    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:42:13.981903 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:13.981876    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:42:15.585046 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:15.585015    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-d989v9bzg"]
Nov 01 05:42:15.586338 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:15.586314    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-d989v9bzg" podUID="98140dfe-532f-416f-8f3b-b549977e7cc1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 05:42:15.750406 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:15.750377    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:42:15.930500 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:15.929775    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:42:16.475314 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:16.475281    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:42:18.632209 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:18.632171    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:42:18.805299 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:18.805264    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:42:20.585498 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:20.585430    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-d989v9bzg" podUID="98140dfe-532f-416f-8f3b-b549977e7cc1" containerName="payload-processing" containerID="cri-o://f3fb80117d24ffeedd4aff63c341c1a6765d8b30cd5a74dcba2b112935633954" gracePeriod=25
Nov 01 05:42:20.790659 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:20.790621    5689 generic.go:356] "Generic (PLEG): container finished" podID="98140dfe-532f-416f-8f3b-b549977e7cc1" containerID="f3fb80117d24ffeedd4aff63c341c1a6765d8b30cd5a74dcba2b112935633954" exitCode=0
Nov 01 05:42:20.790840 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:20.790693    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-d989v9bzg" event={"ID":"98140dfe-532f-416f-8f3b-b549977e7cc1","Type":"ContainerDied","Data":"f3fb80117d24ffeedd4aff63c341c1a6765d8b30cd5a74dcba2b112935633954"}
Nov 01 05:42:20.847414 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:20.847359    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-d989v9bzg"
Nov 01 05:42:20.988783 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:20.988751    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98140dfe-532f-416f-8f3b-b549977e7cc1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98140dfe-532f-416f-8f3b-b549977e7cc1-plugins-config-volume\") pod \"98140dfe-532f-416f-8f3b-b549977e7cc1\" (UID: \"98140dfe-532f-416f-8f3b-b549977e7cc1\") "
Nov 01 05:42:20.988957 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:20.988801    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98140dfe-532f-416f-8f3b-b549977e7cc1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/98140dfe-532f-416f-8f3b-b549977e7cc1-routing-overlay-volume\") pod \"98140dfe-532f-416f-8f3b-b549977e7cc1\" (UID: \"98140dfe-532f-416f-8f3b-b549977e7cc1\") "
Nov 01 05:42:20.989029 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:20.988968    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98140dfe-532f-416f-8f3b-b549977e7cc1-kube-api-access-29k7d\" (UniqueName: \"kubernetes.io/projected/98140dfe-532f-416f-8f3b-b549977e7cc1-kube-api-access-29k7d\") pod \"98140dfe-532f-416f-8f3b-b549977e7cc1\" (UID: \"98140dfe-532f-416f-8f3b-b549977e7cc1\") "
Nov 01 05:42:20.989029 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:20.989008    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98140dfe-532f-416f-8f3b-b549977e7cc1-provider-credentials\" (UniqueName: \"kubernetes.io/projected/98140dfe-532f-416f-8f3b-b549977e7cc1-provider-credentials\") pod \"98140dfe-532f-416f-8f3b-b549977e7cc1\" (UID: \"98140dfe-532f-416f-8f3b-b549977e7cc1\") "
Nov 01 05:42:20.989209 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:20.989183    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98140dfe-532f-416f-8f3b-b549977e7cc1-routing-overlay-volume" pod "98140dfe-532f-416f-8f3b-b549977e7cc1" (UID: "98140dfe-532f-416f-8f3b-b549977e7cc1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:42:20.989271 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:20.989224    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98140dfe-532f-416f-8f3b-b549977e7cc1-plugins-config-volume" pod "98140dfe-532f-416f-8f3b-b549977e7cc1" (UID: "98140dfe-532f-416f-8f3b-b549977e7cc1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:42:20.989412 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:20.989395    5689 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98140dfe-532f-416f-8f3b-b549977e7cc1-plugins-config-volume\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:42:20.989491 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:20.989419    5689 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/98140dfe-532f-416f-8f3b-b549977e7cc1-routing-overlay-volume\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:42:20.991224 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:20.991204    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98140dfe-532f-416f-8f3b-b549977e7cc1-kube-api-access-29k7d" pod "98140dfe-532f-416f-8f3b-b549977e7cc1" (UID: "98140dfe-532f-416f-8f3b-b549977e7cc1"). InnerVolumeSpecName "kube-api-access-29k7d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:42:20.991309 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:20.991291    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98140dfe-532f-416f-8f3b-b549977e7cc1-provider-credentials" pod "98140dfe-532f-416f-8f3b-b549977e7cc1" (UID: "98140dfe-532f-416f-8f3b-b549977e7cc1"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:42:21.090609 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:21.090577    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29k7d\" (UniqueName: \"kubernetes.io/projected/98140dfe-532f-416f-8f3b-b549977e7cc1-kube-api-access-29k7d\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:42:21.090609 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:21.090604    5689 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/98140dfe-532f-416f-8f3b-b549977e7cc1-provider-credentials\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:42:21.797382 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:21.797353    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-d989v9bzg"
Nov 01 05:42:21.797382 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:21.797374    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-d989v9bzg" event={"ID":"98140dfe-532f-416f-8f3b-b549977e7cc1","Type":"ContainerDied","Data":"40f18c24edebfce9f13a15bb561a80050fb636efc06303ff778a243a0c5a4eed"}
Nov 01 05:42:21.797967 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:21.797438    5689 scope.go:122] "RemoveContainer" containerID="f3fb80117d24ffeedd4aff63c341c1a6765d8b30cd5a74dcba2b112935633954"
Nov 01 05:42:21.961283 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:21.961253    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-d989v9bzg"]
Nov 01 05:42:22.010208 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:22.010181    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-c6e99c/payload-processing-external-model-e2e-extmod-b-c6e99c-d989v9bzg"]
Nov 01 05:42:22.068870 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:22.068807    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98140dfe-532f-416f-8f3b-b549977e7cc1" path="/var/lib/kubelet/pods/98140dfe-532f-416f-8f3b-b549977e7cc1/volumes"
Nov 01 05:42:22.562048 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:22.562017    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:42:22.746889 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:22.746852    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:42:23.142848 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:23.142819    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:42:23.335772 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:23.335738    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:42:23.527868 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:23.527830    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:42:23.713475 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:23.713433    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:42:24.322982 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:24.322951    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:42:24.504406 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:24.504378    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:42:26.926676 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:26.926644    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:42:27.311277 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:27.311207    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:42:28.071218 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:28.071153    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:42:28.386802 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:28.386692    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:42:28.755971 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:28.755924    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:42:28.929321 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:28.929288    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:42:29.385379 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:29.385347    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:42:30.069199 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:42:30.069166    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:42:31.390171 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:31.390137    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a08eb536-768db898c5-4m5l2"]
Nov 01 05:42:31.788840 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:31.788809    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:42:31.968284 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:31.968250    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:42:33.722670 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:33.722638    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:42:35.197223 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:35.197193    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:42:36.390976 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:36.390924    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-a08eb536-768db898c5-4m5l2" podUID="9831d79c-480d-4510-9862-dd03e537e768" containerName="payload-processing" containerID="cri-o://c59676e67c376b28415a6ad6ffa1394daf42e77587681743d85d7f8d0dfba54d" gracePeriod=25
Nov 01 05:42:36.664073 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:36.664047    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a08eb536-768db898c5-4m5l2"
Nov 01 05:42:36.711803 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:36.711773    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9831d79c-480d-4510-9862-dd03e537e768-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9831d79c-480d-4510-9862-dd03e537e768-plugins-config-volume\") pod \"9831d79c-480d-4510-9862-dd03e537e768\" (UID: \"9831d79c-480d-4510-9862-dd03e537e768\") "
Nov 01 05:42:36.711949 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:36.711856    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9831d79c-480d-4510-9862-dd03e537e768-kube-api-access-4ll8l\" (UniqueName: \"kubernetes.io/projected/9831d79c-480d-4510-9862-dd03e537e768-kube-api-access-4ll8l\") pod \"9831d79c-480d-4510-9862-dd03e537e768\" (UID: \"9831d79c-480d-4510-9862-dd03e537e768\") "
Nov 01 05:42:36.712173 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:36.712141    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9831d79c-480d-4510-9862-dd03e537e768-plugins-config-volume" pod "9831d79c-480d-4510-9862-dd03e537e768" (UID: "9831d79c-480d-4510-9862-dd03e537e768"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:42:36.713978 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:36.713957    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9831d79c-480d-4510-9862-dd03e537e768-kube-api-access-4ll8l" pod "9831d79c-480d-4510-9862-dd03e537e768" (UID: "9831d79c-480d-4510-9862-dd03e537e768"). InnerVolumeSpecName "kube-api-access-4ll8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:42:36.812848 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:36.812810    5689 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9831d79c-480d-4510-9862-dd03e537e768-plugins-config-volume\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:42:36.812848 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:36.812836    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ll8l\" (UniqueName: \"kubernetes.io/projected/9831d79c-480d-4510-9862-dd03e537e768-kube-api-access-4ll8l\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:42:36.889227 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:36.889196    5689 generic.go:356] "Generic (PLEG): container finished" podID="9831d79c-480d-4510-9862-dd03e537e768" containerID="c59676e67c376b28415a6ad6ffa1394daf42e77587681743d85d7f8d0dfba54d" exitCode=0
Nov 01 05:42:36.889361 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:36.889237    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a08eb536-768db898c5-4m5l2"
Nov 01 05:42:36.889361 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:36.889283    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a08eb536-768db898c5-4m5l2" event={"ID":"9831d79c-480d-4510-9862-dd03e537e768","Type":"ContainerDied","Data":"c59676e67c376b28415a6ad6ffa1394daf42e77587681743d85d7f8d0dfba54d"}
Nov 01 05:42:36.889434 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:36.889359    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a08eb536-768db898c5-4m5l2" event={"ID":"9831d79c-480d-4510-9862-dd03e537e768","Type":"ContainerDied","Data":"27df4138417f9c21c5ce6b61c0a0f55841ec71419f3025868f735a30601abf53"}
Nov 01 05:42:36.889434 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:36.889382    5689 scope.go:122] "RemoveContainer" containerID="c59676e67c376b28415a6ad6ffa1394daf42e77587681743d85d7f8d0dfba54d"
Nov 01 05:42:36.899321 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:36.899301    5689 scope.go:122] "RemoveContainer" containerID="c59676e67c376b28415a6ad6ffa1394daf42e77587681743d85d7f8d0dfba54d"
Nov 01 05:42:36.899623 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:42:36.899601    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c59676e67c376b28415a6ad6ffa1394daf42e77587681743d85d7f8d0dfba54d\": container with ID starting with c59676e67c376b28415a6ad6ffa1394daf42e77587681743d85d7f8d0dfba54d not found: ID does not exist" containerID="c59676e67c376b28415a6ad6ffa1394daf42e77587681743d85d7f8d0dfba54d"
Nov 01 05:42:36.899702 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:36.899631    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c59676e67c376b28415a6ad6ffa1394daf42e77587681743d85d7f8d0dfba54d"} err="failed to get container status \"c59676e67c376b28415a6ad6ffa1394daf42e77587681743d85d7f8d0dfba54d\": rpc error: code = NotFound desc = could not find container \"c59676e67c376b28415a6ad6ffa1394daf42e77587681743d85d7f8d0dfba54d\": container with ID starting with c59676e67c376b28415a6ad6ffa1394daf42e77587681743d85d7f8d0dfba54d not found: ID does not exist"
Nov 01 05:42:37.052607 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:37.052579    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a08eb536-768db898c5-4m5l2"]
Nov 01 05:42:37.056859 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:37.056833    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a08eb536-768db898c5-4m5l2"]
Nov 01 05:42:38.067432 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:38.067402    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9831d79c-480d-4510-9862-dd03e537e768" path="/var/lib/kubelet/pods/9831d79c-480d-4510-9862-dd03e537e768/volumes"
Nov 01 05:42:38.501211 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:38.501177    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:42:38.715927 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:38.715896    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:42:43.592892 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:43.592859    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:42:47.982731 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:47.982697    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a08eb536-6465b89f5b-xgr2d"]
Nov 01 05:42:47.983105 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:47.982931    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a08eb536-6465b89f5b-xgr2d" podUID="e0edbadf-884c-409f-9ce4-5b4182df2775" containerName="maas-api" containerID="cri-o://528634dfff97f941aa8f639d67d8a31718e9c711a0ed1a0e677d0e7b63bf9c97" gracePeriod=30
Nov 01 05:42:48.130340 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:48.130296    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:42:48.276301 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:48.276278    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a08eb536-6465b89f5b-xgr2d"
Nov 01 05:42:48.344635 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:48.344610    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0edbadf-884c-409f-9ce4-5b4182df2775-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0edbadf-884c-409f-9ce4-5b4182df2775-maas-api-tls\") pod \"e0edbadf-884c-409f-9ce4-5b4182df2775\" (UID: \"e0edbadf-884c-409f-9ce4-5b4182df2775\") "
Nov 01 05:42:48.344802 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:48.344692    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0edbadf-884c-409f-9ce4-5b4182df2775-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0edbadf-884c-409f-9ce4-5b4182df2775-openshift-service-ca\") pod \"e0edbadf-884c-409f-9ce4-5b4182df2775\" (UID: \"e0edbadf-884c-409f-9ce4-5b4182df2775\") "
Nov 01 05:42:48.344877 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:48.344802    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0edbadf-884c-409f-9ce4-5b4182df2775-kube-api-access-dhzbq\" (UniqueName: \"kubernetes.io/projected/e0edbadf-884c-409f-9ce4-5b4182df2775-kube-api-access-dhzbq\") pod \"e0edbadf-884c-409f-9ce4-5b4182df2775\" (UID: \"e0edbadf-884c-409f-9ce4-5b4182df2775\") "
Nov 01 05:42:48.344877 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:48.344862    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0edbadf-884c-409f-9ce4-5b4182df2775-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0edbadf-884c-409f-9ce4-5b4182df2775-metrics-certs\") pod \"e0edbadf-884c-409f-9ce4-5b4182df2775\" (UID: \"e0edbadf-884c-409f-9ce4-5b4182df2775\") "
Nov 01 05:42:48.345031 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:48.345006    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0edbadf-884c-409f-9ce4-5b4182df2775-openshift-service-ca" pod "e0edbadf-884c-409f-9ce4-5b4182df2775" (UID: "e0edbadf-884c-409f-9ce4-5b4182df2775"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:42:48.345260 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:48.345242    5689 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0edbadf-884c-409f-9ce4-5b4182df2775-openshift-service-ca\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:42:48.346991 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:48.346963    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0edbadf-884c-409f-9ce4-5b4182df2775-kube-api-access-dhzbq" pod "e0edbadf-884c-409f-9ce4-5b4182df2775" (UID: "e0edbadf-884c-409f-9ce4-5b4182df2775"). InnerVolumeSpecName "kube-api-access-dhzbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:42:48.347100 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:48.347015    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0edbadf-884c-409f-9ce4-5b4182df2775-maas-api-tls" pod "e0edbadf-884c-409f-9ce4-5b4182df2775" (UID: "e0edbadf-884c-409f-9ce4-5b4182df2775"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 05:42:48.347100 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:48.347036    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0edbadf-884c-409f-9ce4-5b4182df2775-metrics-certs" pod "e0edbadf-884c-409f-9ce4-5b4182df2775" (UID: "e0edbadf-884c-409f-9ce4-5b4182df2775"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 05:42:48.446505 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:48.446472    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhzbq\" (UniqueName: \"kubernetes.io/projected/e0edbadf-884c-409f-9ce4-5b4182df2775-kube-api-access-dhzbq\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:42:48.446505 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:48.446498    5689 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0edbadf-884c-409f-9ce4-5b4182df2775-metrics-certs\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:42:48.446505 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:48.446508    5689 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0edbadf-884c-409f-9ce4-5b4182df2775-maas-api-tls\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:42:48.966556 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:48.966523    5689 generic.go:356] "Generic (PLEG): container finished" podID="e0edbadf-884c-409f-9ce4-5b4182df2775" containerID="528634dfff97f941aa8f639d67d8a31718e9c711a0ed1a0e677d0e7b63bf9c97" exitCode=0
Nov 01 05:42:48.966752 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:48.966573    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a08eb536-6465b89f5b-xgr2d"
Nov 01 05:42:48.966752 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:48.966603    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a08eb536-6465b89f5b-xgr2d" event={"ID":"e0edbadf-884c-409f-9ce4-5b4182df2775","Type":"ContainerDied","Data":"528634dfff97f941aa8f639d67d8a31718e9c711a0ed1a0e677d0e7b63bf9c97"}
Nov 01 05:42:48.966752 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:48.966656    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a08eb536-6465b89f5b-xgr2d" event={"ID":"e0edbadf-884c-409f-9ce4-5b4182df2775","Type":"ContainerDied","Data":"4e0088f658bd0dc590301fca13be41822e91be3683b36b46501eb1d06049973a"}
Nov 01 05:42:48.966752 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:48.966678    5689 scope.go:122] "RemoveContainer" containerID="528634dfff97f941aa8f639d67d8a31718e9c711a0ed1a0e677d0e7b63bf9c97"
Nov 01 05:42:48.989402 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:48.989377    5689 scope.go:122] "RemoveContainer" containerID="528634dfff97f941aa8f639d67d8a31718e9c711a0ed1a0e677d0e7b63bf9c97"
Nov 01 05:42:48.992213 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:42:48.992183    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"528634dfff97f941aa8f639d67d8a31718e9c711a0ed1a0e677d0e7b63bf9c97\": container with ID starting with 528634dfff97f941aa8f639d67d8a31718e9c711a0ed1a0e677d0e7b63bf9c97 not found: ID does not exist" containerID="528634dfff97f941aa8f639d67d8a31718e9c711a0ed1a0e677d0e7b63bf9c97"
Nov 01 05:42:48.992380 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:48.992354    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"528634dfff97f941aa8f639d67d8a31718e9c711a0ed1a0e677d0e7b63bf9c97"} err="failed to get container status \"528634dfff97f941aa8f639d67d8a31718e9c711a0ed1a0e677d0e7b63bf9c97\": rpc error: code = NotFound desc = could not find container \"528634dfff97f941aa8f639d67d8a31718e9c711a0ed1a0e677d0e7b63bf9c97\": container with ID starting with 528634dfff97f941aa8f639d67d8a31718e9c711a0ed1a0e677d0e7b63bf9c97 not found: ID does not exist"
Nov 01 05:42:49.074314 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:49.074285    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:42:49.136484 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:49.136225    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a08eb536-6465b89f5b-xgr2d"]
Nov 01 05:42:49.140656 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:49.140618    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a08eb536-6465b89f5b-xgr2d"]
Nov 01 05:42:49.808169 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:49.808127    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:42:49.988186 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:49.988146    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:42:50.067973 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:50.067878    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0edbadf-884c-409f-9ce4-5b4182df2775" path="/var/lib/kubelet/pods/e0edbadf-884c-409f-9ce4-5b4182df2775/volumes"
Nov 01 05:42:50.273318 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:50.273282    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd"]
Nov 01 05:42:50.273905 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:50.273885    5689 cpu_manager.go:414] "containerMap: removing container" podUID="9831d79c-480d-4510-9862-dd03e537e768" containerName="payload-processing"
Nov 01 05:42:50.274030 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:50.273906    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9831d79c-480d-4510-9862-dd03e537e768" containerName="payload-processing"
Nov 01 05:42:50.274030 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:50.273975    5689 cpu_manager.go:414] "containerMap: removing container" podUID="98140dfe-532f-416f-8f3b-b549977e7cc1" containerName="payload-processing"
Nov 01 05:42:50.274030 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:50.273986    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98140dfe-532f-416f-8f3b-b549977e7cc1" containerName="payload-processing"
Nov 01 05:42:50.274231 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:50.274045    5689 cpu_manager.go:414] "containerMap: removing container" podUID="c4daddd7-ac26-4180-8a37-c273bc6c9f6d" containerName="payload-processing"
Nov 01 05:42:50.274231 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:50.274053    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4daddd7-ac26-4180-8a37-c273bc6c9f6d" containerName="payload-processing"
Nov 01 05:42:50.274231 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:50.274069    5689 cpu_manager.go:414] "containerMap: removing container" podUID="e0edbadf-884c-409f-9ce4-5b4182df2775" containerName="maas-api"
Nov 01 05:42:50.274231 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:50.274078    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0edbadf-884c-409f-9ce4-5b4182df2775" containerName="maas-api"
Nov 01 05:42:50.274231 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:50.274215    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="98140dfe-532f-416f-8f3b-b549977e7cc1" containerName="payload-processing"
Nov 01 05:42:50.274506 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:50.274238    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4daddd7-ac26-4180-8a37-c273bc6c9f6d" containerName="payload-processing"
Nov 01 05:42:50.274506 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:50.274251    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0edbadf-884c-409f-9ce4-5b4182df2775" containerName="maas-api"
Nov 01 05:42:50.274506 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:50.274262    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="9831d79c-480d-4510-9862-dd03e537e768" containerName="payload-processing"
Nov 01 05:42:50.278837 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:50.278817    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd"
Nov 01 05:42:50.335529 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:50.332630    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-8815d2-openshift-default-dockercfg-9jslz\""
Nov 01 05:42:50.408359 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:50.408322    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e28aa73d-02c7-4b5f-9d59-fca29e73cc89-workload-socket\") pod \"e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd\" (UID: \"e28aa73d-02c7-4b5f-9d59-fca29e73cc89\") " pod="openshift-ingress/e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd"
Nov 01 05:42:50.408718 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:50.408696    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6xwv\" (UniqueName: \"kubernetes.io/projected/e28aa73d-02c7-4b5f-9d59-fca29e73cc89-kube-api-access-b6xwv\") pod \"e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd\" (UID: \"e28aa73d-02c7-4b5f-9d59-fca29e73cc89\") " pod="openshift-ingress/e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd"
Nov 01 05:42:50.408929 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:50.408911    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e28aa73d-02c7-4b5f-9d59-fca29e73cc89-istio-podinfo\") pod \"e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd\" (UID: \"e28aa73d-02c7-4b5f-9d59-fca29e73cc89\") " pod="openshift-ingress/e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd"
Nov 01 05:42:50.409045 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:50.409030    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e28aa73d-02c7-4b5f-9d59-fca29e73cc89-istiod-ca-cert\") pod \"e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd\" (UID: \"e28aa73d-02c7-4b5f-9d59-fca29e73cc89\") " pod="openshift-ingress/e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd"
Nov 01 05:42:50.409194 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:50.409175    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e28aa73d-02c7-4b5f-9d59-fca29e73cc89-istio-envoy\") pod \"e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd\" (UID: \"e28aa73d-02c7-4b5f-9d59-fca29e73cc89\") " pod="openshift-ingress/e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd"
Nov 01 05:42:50.409398 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:50.409378    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e28aa73d-02c7-4b5f-9d59-fca29e73cc89-credential-socket\") pod \"e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd\" (UID: \"e28aa73d-02c7-4b5f-9d59-fca29e73cc89\") " pod="openshift-ingress/e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd"
Nov 01 05:42:50.410388 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:50.410367    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e28aa73d-02c7-4b5f-9d59-fca29e73cc89-workload-certs\") pod \"e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd\" (UID: \"e28aa73d-02c7-4b5f-9d59-fca29e73cc89\") " pod="openshift-ingress/e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd"
Nov 01 05:42:50.410688 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:50.410665    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e28aa73d-02c7-4b5f-9d59-fca29e73cc89-istio-data\") pod \"e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd\" (UID: \"e28aa73d-02c7-4b5f-9d59-fca29e73cc89\") " pod="openshift-ingress/e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd"
Nov 01 05:42:50.410802 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:50.410787    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e28aa73d-02c7-4b5f-9d59-fca29e73cc89-istio-token\") pod \"e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd\" (UID: \"e28aa73d-02c7-4b5f-9d59-fca29e73cc89\") " pod="openshift-ingress/e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd"
Nov 01 05:42:50.448602 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:50.447494    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:42:50.464287 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:50.463505    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd"]
Nov 01 05:42:50.512414 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:50.512384    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e28aa73d-02c7-4b5f-9d59-fca29e73cc89-istio-data\") pod \"e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd\" (UID: \"e28aa73d-02c7-4b5f-9d59-fca29e73cc89\") " pod="openshift-ingress/e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd"
Nov 01 05:42:50.512414 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:50.512417    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e28aa73d-02c7-4b5f-9d59-fca29e73cc89-istio-token\") pod \"e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd\" (UID: \"e28aa73d-02c7-4b5f-9d59-fca29e73cc89\") " pod="openshift-ingress/e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd"
Nov 01 05:42:50.512704 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:50.512473    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e28aa73d-02c7-4b5f-9d59-fca29e73cc89-workload-socket\") pod \"e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd\" (UID: \"e28aa73d-02c7-4b5f-9d59-fca29e73cc89\") " pod="openshift-ingress/e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd"
Nov 01 05:42:50.512704 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:50.512511    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6xwv\" (UniqueName: \"kubernetes.io/projected/e28aa73d-02c7-4b5f-9d59-fca29e73cc89-kube-api-access-b6xwv\") pod \"e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd\" (UID: \"e28aa73d-02c7-4b5f-9d59-fca29e73cc89\") " pod="openshift-ingress/e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd"
Nov 01 05:42:50.512704 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:50.512562    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e28aa73d-02c7-4b5f-9d59-fca29e73cc89-istio-podinfo\") pod \"e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd\" (UID: \"e28aa73d-02c7-4b5f-9d59-fca29e73cc89\") " pod="openshift-ingress/e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd"
Nov 01 05:42:50.512866 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:50.512791    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e28aa73d-02c7-4b5f-9d59-fca29e73cc89-istio-data\") pod \"e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd\" (UID: \"e28aa73d-02c7-4b5f-9d59-fca29e73cc89\") " pod="openshift-ingress/e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd"
Nov 01 05:42:50.512971 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:50.512950    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e28aa73d-02c7-4b5f-9d59-fca29e73cc89-workload-socket\") pod \"e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd\" (UID: \"e28aa73d-02c7-4b5f-9d59-fca29e73cc89\") " pod="openshift-ingress/e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd"
Nov 01 05:42:50.513021 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:50.513005    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e28aa73d-02c7-4b5f-9d59-fca29e73cc89-istiod-ca-cert\") pod \"e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd\" (UID: \"e28aa73d-02c7-4b5f-9d59-fca29e73cc89\") " pod="openshift-ingress/e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd"
Nov 01 05:42:50.513072 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:50.513058    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e28aa73d-02c7-4b5f-9d59-fca29e73cc89-istio-envoy\") pod \"e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd\" (UID: \"e28aa73d-02c7-4b5f-9d59-fca29e73cc89\") " pod="openshift-ingress/e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd"
Nov 01 05:42:50.513149 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:50.513134    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e28aa73d-02c7-4b5f-9d59-fca29e73cc89-credential-socket\") pod \"e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd\" (UID: \"e28aa73d-02c7-4b5f-9d59-fca29e73cc89\") " pod="openshift-ingress/e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd"
Nov 01 05:42:50.513215 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:50.513163    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e28aa73d-02c7-4b5f-9d59-fca29e73cc89-workload-certs\") pod \"e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd\" (UID: \"e28aa73d-02c7-4b5f-9d59-fca29e73cc89\") " pod="openshift-ingress/e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd"
Nov 01 05:42:50.513371 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:50.513352    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e28aa73d-02c7-4b5f-9d59-fca29e73cc89-istiod-ca-cert\") pod \"e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd\" (UID: \"e28aa73d-02c7-4b5f-9d59-fca29e73cc89\") " pod="openshift-ingress/e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd"
Nov 01 05:42:50.513639 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:50.513430    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e28aa73d-02c7-4b5f-9d59-fca29e73cc89-credential-socket\") pod \"e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd\" (UID: \"e28aa73d-02c7-4b5f-9d59-fca29e73cc89\") " pod="openshift-ingress/e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd"
Nov 01 05:42:50.513639 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:50.513520    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e28aa73d-02c7-4b5f-9d59-fca29e73cc89-workload-certs\") pod \"e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd\" (UID: \"e28aa73d-02c7-4b5f-9d59-fca29e73cc89\") " pod="openshift-ingress/e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd"
Nov 01 05:42:50.515307 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:50.515285    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e28aa73d-02c7-4b5f-9d59-fca29e73cc89-istio-envoy\") pod \"e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd\" (UID: \"e28aa73d-02c7-4b5f-9d59-fca29e73cc89\") " pod="openshift-ingress/e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd"
Nov 01 05:42:50.515406 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:50.515344    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e28aa73d-02c7-4b5f-9d59-fca29e73cc89-istio-podinfo\") pod \"e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd\" (UID: \"e28aa73d-02c7-4b5f-9d59-fca29e73cc89\") " pod="openshift-ingress/e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd"
Nov 01 05:42:50.573335 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:50.573307    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e28aa73d-02c7-4b5f-9d59-fca29e73cc89-istio-token\") pod \"e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd\" (UID: \"e28aa73d-02c7-4b5f-9d59-fca29e73cc89\") " pod="openshift-ingress/e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd"
Nov 01 05:42:50.573524 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:50.573506    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6xwv\" (UniqueName: \"kubernetes.io/projected/e28aa73d-02c7-4b5f-9d59-fca29e73cc89-kube-api-access-b6xwv\") pod \"e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd\" (UID: \"e28aa73d-02c7-4b5f-9d59-fca29e73cc89\") " pod="openshift-ingress/e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd"
Nov 01 05:42:50.592526 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:50.592478    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd"
Nov 01 05:42:50.734606 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:50.734571    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:42:50.980577 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:50.980547    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:42:51.024409 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:51.024383    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd"]
Nov 01 05:42:51.027357 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:42:51.027331    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode28aa73d_02c7_4b5f_9d59_fca29e73cc89.slice/crio-724bc27a35fa490305b50917cd369b66b5ffc06dd8300c5181e433fae3492d30 WatchSource:0}: Error finding container 724bc27a35fa490305b50917cd369b66b5ffc06dd8300c5181e433fae3492d30: Status 404 returned error can't find the container with id 724bc27a35fa490305b50917cd369b66b5ffc06dd8300c5181e433fae3492d30
Nov 01 05:42:51.029739 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:51.029708    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:42:51.029815 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:51.029770    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:42:51.029815 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:51.029800    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:42:51.985337 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:51.985297    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd" event={"ID":"e28aa73d-02c7-4b5f-9d59-fca29e73cc89","Type":"ContainerStarted","Data":"474e0d5e5db9511ac78c298d96bb5f69b8446299a7edcb72b7c426b8cb722819"}
Nov 01 05:42:51.985814 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:51.985390    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd" event={"ID":"e28aa73d-02c7-4b5f-9d59-fca29e73cc89","Type":"ContainerStarted","Data":"724bc27a35fa490305b50917cd369b66b5ffc06dd8300c5181e433fae3492d30"}
Nov 01 05:42:52.154998 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:52.154942    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd" podStartSLOduration=2.154928322 podStartE2EDuration="2.154928322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:42:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:42:52.153291951 +0000 UTC m=+2350.224088697" watchObservedRunningTime="2026-11-01 05:42:52.154928322 +0000 UTC m=+2350.225725116"
Nov 01 05:42:52.593202 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:52.593170    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd"
Nov 01 05:42:52.598658 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:52.598633    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd"
Nov 01 05:42:52.990955 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:52.990924    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd"
Nov 01 05:42:52.991919 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:52.991894    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd"
Nov 01 05:42:55.048031 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:55.048004    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:42:55.221178 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:55.221145    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:42:55.413137 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:55.413107    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:42:58.287305 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:58.287278    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:42:58.460209 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:58.460182    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:42:58.819447 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:58.819412    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8815d2-58457b5f85-t72rh"]
Nov 01 05:42:58.825257 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:58.825139    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8815d2-58457b5f85-t72rh"
Nov 01 05:42:58.873292 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:58.873263    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8815d2-58457b5f85-t72rh"]
Nov 01 05:42:58.916903 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:58.916873    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7j77\" (UniqueName: \"kubernetes.io/projected/6fcd6eef-c926-4e83-a83e-e14e2a6ec6e7-kube-api-access-f7j77\") pod \"payload-processing-e2e-api-b-8815d2-58457b5f85-t72rh\" (UID: \"6fcd6eef-c926-4e83-a83e-e14e2a6ec6e7\") " pod="openshift-ingress/payload-processing-e2e-api-b-8815d2-58457b5f85-t72rh"
Nov 01 05:42:58.917064 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:58.917048    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fcd6eef-c926-4e83-a83e-e14e2a6ec6e7-plugins-config-volume\") pod \"payload-processing-e2e-api-b-8815d2-58457b5f85-t72rh\" (UID: \"6fcd6eef-c926-4e83-a83e-e14e2a6ec6e7\") " pod="openshift-ingress/payload-processing-e2e-api-b-8815d2-58457b5f85-t72rh"
Nov 01 05:42:58.919790 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:58.919770    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-8815d2\""
Nov 01 05:42:58.919790 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:58.919768    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-8815d2-dockercfg-rwv6p\""
Nov 01 05:42:58.948644 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:58.948613    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:42:59.018389 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:59.018343    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fcd6eef-c926-4e83-a83e-e14e2a6ec6e7-plugins-config-volume\") pod \"payload-processing-e2e-api-b-8815d2-58457b5f85-t72rh\" (UID: \"6fcd6eef-c926-4e83-a83e-e14e2a6ec6e7\") " pod="openshift-ingress/payload-processing-e2e-api-b-8815d2-58457b5f85-t72rh"
Nov 01 05:42:59.018602 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:59.018482    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7j77\" (UniqueName: \"kubernetes.io/projected/6fcd6eef-c926-4e83-a83e-e14e2a6ec6e7-kube-api-access-f7j77\") pod \"payload-processing-e2e-api-b-8815d2-58457b5f85-t72rh\" (UID: \"6fcd6eef-c926-4e83-a83e-e14e2a6ec6e7\") " pod="openshift-ingress/payload-processing-e2e-api-b-8815d2-58457b5f85-t72rh"
Nov 01 05:42:59.019069 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:59.019051    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fcd6eef-c926-4e83-a83e-e14e2a6ec6e7-plugins-config-volume\") pod \"payload-processing-e2e-api-b-8815d2-58457b5f85-t72rh\" (UID: \"6fcd6eef-c926-4e83-a83e-e14e2a6ec6e7\") " pod="openshift-ingress/payload-processing-e2e-api-b-8815d2-58457b5f85-t72rh"
Nov 01 05:42:59.079176 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:59.079111    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7j77\" (UniqueName: \"kubernetes.io/projected/6fcd6eef-c926-4e83-a83e-e14e2a6ec6e7-kube-api-access-f7j77\") pod \"payload-processing-e2e-api-b-8815d2-58457b5f85-t72rh\" (UID: \"6fcd6eef-c926-4e83-a83e-e14e2a6ec6e7\") " pod="openshift-ingress/payload-processing-e2e-api-b-8815d2-58457b5f85-t72rh"
Nov 01 05:42:59.126345 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:59.126320    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:42:59.151168 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:59.151139    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8815d2-58457b5f85-t72rh"
Nov 01 05:42:59.528939 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:42:59.528917    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8815d2-58457b5f85-t72rh"]
Nov 01 05:42:59.531918 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:42:59.531888    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fcd6eef_c926_4e83_a83e_e14e2a6ec6e7.slice/crio-ac3d9f6e4e8a8b3243df9a938c8db406f08920f9a8ae9f68ccd62b2f3eea8cde WatchSource:0}: Error finding container ac3d9f6e4e8a8b3243df9a938c8db406f08920f9a8ae9f68ccd62b2f3eea8cde: Status 404 returned error can't find the container with id ac3d9f6e4e8a8b3243df9a938c8db406f08920f9a8ae9f68ccd62b2f3eea8cde
Nov 01 05:43:00.031080 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:00.031041    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8815d2-58457b5f85-t72rh" event={"ID":"6fcd6eef-c926-4e83-a83e-e14e2a6ec6e7","Type":"ContainerStarted","Data":"ffd62110ca7bb0ca73e52ae08d91c57cabb01e291da15e7f78bc1b0efa592ad4"}
Nov 01 05:43:00.031262 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:00.031103    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8815d2-58457b5f85-t72rh" event={"ID":"6fcd6eef-c926-4e83-a83e-e14e2a6ec6e7","Type":"ContainerStarted","Data":"ac3d9f6e4e8a8b3243df9a938c8db406f08920f9a8ae9f68ccd62b2f3eea8cde"}
Nov 01 05:43:00.031321 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:00.031274    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-8815d2-58457b5f85-t72rh"
Nov 01 05:43:00.069755 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:43:00.069719    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:43:00.147802 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:00.147753    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-8815d2-58457b5f85-t72rh" podStartSLOduration=2.1477387390000002 podStartE2EDuration="2.147738739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:42:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:43:00.144926454 +0000 UTC m=+2358.215723174" watchObservedRunningTime="2026-11-01 05:43:00.147738739 +0000 UTC m=+2358.218535456"
Nov 01 05:43:00.356724 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:00.356645    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:43:00.538374 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:00.538342    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:43:00.945239 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:00.945205    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:43:01.444574 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:01.444541    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:43:01.655654 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:01.655621    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:43:01.875143 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:01.875062    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:43:05.703907 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:05.703869    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:43:07.062659 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:07.062628    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:43:07.932446 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:07.932412    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:43:08.121754 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:08.121725    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:43:08.521622 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:08.521591    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:43:10.898440 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:10.898396    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:43:11.083208 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:11.083183    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-8815d2-58457b5f85-t72rh"
Nov 01 05:43:14.838536 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:14.838503    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:43:15.012035 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:15.012001    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:43:15.829026 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:15.828994    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:43:20.611084 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:20.611045    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:43:21.945400 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:21.945367    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ad90c52d-d779bc747-ngwgg"]
Nov 01 05:43:22.049558 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:22.049525    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ad90c52d-7b64b785f-kn8k5"]
Nov 01 05:43:22.050574 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:22.050546    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:43:22.284813 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:22.284739    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:43:22.477943 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:22.477174    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:43:22.652010 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:22.651973    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:43:24.219913 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:24.219882    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:43:25.466388 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:25.466357    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:43:26.886904 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:26.886865    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:43:26.945792 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:26.945766    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-ad90c52d-d779bc747-ngwgg" podUID="9118c67d-7bca-44ad-afc8-102cb132e193" containerName="payload-processing" containerID="cri-o://eaa5877ddc6c2fa25107eee78638e1a38e37d62b3a04992006c8956d36f1e2fa" gracePeriod=25
Nov 01 05:43:27.050796 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:27.050741    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-ad90c52d-7b64b785f-kn8k5" podUID="e5664a5e-da43-4b3b-8d52-3789fca3d216" containerName="payload-pre-processing" containerID="cri-o://b5b39842f4ee53af50d7851b04c957b0897bfd3b6d26f4dc1a4417ff6e11f7a0" gracePeriod=25
Nov 01 05:43:27.072730 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:27.072702    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:43:27.209429 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:27.209400    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ad90c52d-d779bc747-ngwgg"
Nov 01 05:43:27.226147 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:27.226119    5689 generic.go:356] "Generic (PLEG): container finished" podID="9118c67d-7bca-44ad-afc8-102cb132e193" containerID="eaa5877ddc6c2fa25107eee78638e1a38e37d62b3a04992006c8956d36f1e2fa" exitCode=0
Nov 01 05:43:27.226296 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:27.226180    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ad90c52d-d779bc747-ngwgg"
Nov 01 05:43:27.226296 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:27.226209    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ad90c52d-d779bc747-ngwgg" event={"ID":"9118c67d-7bca-44ad-afc8-102cb132e193","Type":"ContainerDied","Data":"eaa5877ddc6c2fa25107eee78638e1a38e37d62b3a04992006c8956d36f1e2fa"}
Nov 01 05:43:27.226399 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:27.226307    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ad90c52d-d779bc747-ngwgg" event={"ID":"9118c67d-7bca-44ad-afc8-102cb132e193","Type":"ContainerDied","Data":"75946495d18ce41d664ad148307a11ea9f3278e931e2c10382e683e37a6d33f6"}
Nov 01 05:43:27.226399 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:27.226338    5689 scope.go:122] "RemoveContainer" containerID="eaa5877ddc6c2fa25107eee78638e1a38e37d62b3a04992006c8956d36f1e2fa"
Nov 01 05:43:27.227957 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:27.227895    5689 generic.go:356] "Generic (PLEG): container finished" podID="e5664a5e-da43-4b3b-8d52-3789fca3d216" containerID="b5b39842f4ee53af50d7851b04c957b0897bfd3b6d26f4dc1a4417ff6e11f7a0" exitCode=0
Nov 01 05:43:27.228107 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:27.228065    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ad90c52d-7b64b785f-kn8k5" event={"ID":"e5664a5e-da43-4b3b-8d52-3789fca3d216","Type":"ContainerDied","Data":"b5b39842f4ee53af50d7851b04c957b0897bfd3b6d26f4dc1a4417ff6e11f7a0"}
Nov 01 05:43:27.236103 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:27.236076    5689 scope.go:122] "RemoveContainer" containerID="eaa5877ddc6c2fa25107eee78638e1a38e37d62b3a04992006c8956d36f1e2fa"
Nov 01 05:43:27.236348 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:43:27.236328    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaa5877ddc6c2fa25107eee78638e1a38e37d62b3a04992006c8956d36f1e2fa\": container with ID starting with eaa5877ddc6c2fa25107eee78638e1a38e37d62b3a04992006c8956d36f1e2fa not found: ID does not exist" containerID="eaa5877ddc6c2fa25107eee78638e1a38e37d62b3a04992006c8956d36f1e2fa"
Nov 01 05:43:27.236408 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:27.236359    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaa5877ddc6c2fa25107eee78638e1a38e37d62b3a04992006c8956d36f1e2fa"} err="failed to get container status \"eaa5877ddc6c2fa25107eee78638e1a38e37d62b3a04992006c8956d36f1e2fa\": rpc error: code = NotFound desc = could not find container \"eaa5877ddc6c2fa25107eee78638e1a38e37d62b3a04992006c8956d36f1e2fa\": container with ID starting with eaa5877ddc6c2fa25107eee78638e1a38e37d62b3a04992006c8956d36f1e2fa not found: ID does not exist"
Nov 01 05:43:27.306377 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:27.306351    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:43:27.316120 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:27.316099    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9118c67d-7bca-44ad-afc8-102cb132e193-kube-api-access-hwfgj\" (UniqueName: \"kubernetes.io/projected/9118c67d-7bca-44ad-afc8-102cb132e193-kube-api-access-hwfgj\") pod \"9118c67d-7bca-44ad-afc8-102cb132e193\" (UID: \"9118c67d-7bca-44ad-afc8-102cb132e193\") "
Nov 01 05:43:27.316261 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:27.316247    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9118c67d-7bca-44ad-afc8-102cb132e193-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9118c67d-7bca-44ad-afc8-102cb132e193-plugins-config-volume\") pod \"9118c67d-7bca-44ad-afc8-102cb132e193\" (UID: \"9118c67d-7bca-44ad-afc8-102cb132e193\") "
Nov 01 05:43:27.316863 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:27.316670    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9118c67d-7bca-44ad-afc8-102cb132e193-plugins-config-volume" pod "9118c67d-7bca-44ad-afc8-102cb132e193" (UID: "9118c67d-7bca-44ad-afc8-102cb132e193"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:43:27.318170 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:27.318149    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9118c67d-7bca-44ad-afc8-102cb132e193-kube-api-access-hwfgj" pod "9118c67d-7bca-44ad-afc8-102cb132e193" (UID: "9118c67d-7bca-44ad-afc8-102cb132e193"). InnerVolumeSpecName "kube-api-access-hwfgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:43:27.333501 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:27.333481    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ad90c52d-7b64b785f-kn8k5"
Nov 01 05:43:27.417622 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:27.417592    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5664a5e-da43-4b3b-8d52-3789fca3d216-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5664a5e-da43-4b3b-8d52-3789fca3d216-plugins-config-volume\") pod \"e5664a5e-da43-4b3b-8d52-3789fca3d216\" (UID: \"e5664a5e-da43-4b3b-8d52-3789fca3d216\") "
Nov 01 05:43:27.417778 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:27.417694    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5664a5e-da43-4b3b-8d52-3789fca3d216-kube-api-access-9mh8g\" (UniqueName: \"kubernetes.io/projected/e5664a5e-da43-4b3b-8d52-3789fca3d216-kube-api-access-9mh8g\") pod \"e5664a5e-da43-4b3b-8d52-3789fca3d216\" (UID: \"e5664a5e-da43-4b3b-8d52-3789fca3d216\") "
Nov 01 05:43:27.417999 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:27.417976    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5664a5e-da43-4b3b-8d52-3789fca3d216-plugins-config-volume" pod "e5664a5e-da43-4b3b-8d52-3789fca3d216" (UID: "e5664a5e-da43-4b3b-8d52-3789fca3d216"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:43:27.418039 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:27.417984    5689 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9118c67d-7bca-44ad-afc8-102cb132e193-plugins-config-volume\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:43:27.418039 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:27.418028    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwfgj\" (UniqueName: \"kubernetes.io/projected/9118c67d-7bca-44ad-afc8-102cb132e193-kube-api-access-hwfgj\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:43:27.419847 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:27.419825    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5664a5e-da43-4b3b-8d52-3789fca3d216-kube-api-access-9mh8g" pod "e5664a5e-da43-4b3b-8d52-3789fca3d216" (UID: "e5664a5e-da43-4b3b-8d52-3789fca3d216"). InnerVolumeSpecName "kube-api-access-9mh8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:43:27.519312 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:27.519240    5689 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5664a5e-da43-4b3b-8d52-3789fca3d216-plugins-config-volume\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:43:27.519312 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:27.519265    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mh8g\" (UniqueName: \"kubernetes.io/projected/e5664a5e-da43-4b3b-8d52-3789fca3d216-kube-api-access-9mh8g\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:43:27.646969 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:27.646938    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ad90c52d-d779bc747-ngwgg"]
Nov 01 05:43:27.653758 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:27.653734    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ad90c52d-d779bc747-ngwgg"]
Nov 01 05:43:28.067770 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:28.067734    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9118c67d-7bca-44ad-afc8-102cb132e193" path="/var/lib/kubelet/pods/9118c67d-7bca-44ad-afc8-102cb132e193/volumes"
Nov 01 05:43:28.236743 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:28.236662    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ad90c52d-7b64b785f-kn8k5" event={"ID":"e5664a5e-da43-4b3b-8d52-3789fca3d216","Type":"ContainerDied","Data":"69947a328c80cdb62b389545bef65c6018d7cebcd2b5718f4d8f8515963603be"}
Nov 01 05:43:28.236743 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:28.236677    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ad90c52d-7b64b785f-kn8k5"
Nov 01 05:43:28.236743 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:28.236711    5689 scope.go:122] "RemoveContainer" containerID="b5b39842f4ee53af50d7851b04c957b0897bfd3b6d26f4dc1a4417ff6e11f7a0"
Nov 01 05:43:28.399714 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:28.399684    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ad90c52d-7b64b785f-kn8k5"]
Nov 01 05:43:28.448974 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:28.448941    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ad90c52d-7b64b785f-kn8k5"]
Nov 01 05:43:28.801996 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:28.801966    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:43:30.068952 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:43:30.068913    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:43:30.068952 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:30.068919    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5664a5e-da43-4b3b-8d52-3789fca3d216" path="/var/lib/kubelet/pods/e5664a5e-da43-4b3b-8d52-3789fca3d216/volumes"
Nov 01 05:43:30.097933 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:30.097911    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:43:30.287804 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:30.287771    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:43:30.931325 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:30.931293    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:43:33.828373 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:33.828339    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:43:34.137076 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:34.136974    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:43:34.828985 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:34.828950    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:43:35.738099 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:35.738067    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:43:40.519387 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:40.519344    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:43:40.723120 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:40.723084    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:43:40.900325 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:40.900286    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:43:45.310645 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:45.310615    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:43:49.720100 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:49.720070    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:43:49.898022 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:49.897979    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:43:50.104439 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:50.104363    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:43:50.902307 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:50.902276    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:43:54.605746 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:54.605719    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8815d2-7c78d444b-x7v7h"]
Nov 01 05:43:54.606297 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:54.606283    5689 cpu_manager.go:414] "containerMap: removing container" podUID="9118c67d-7bca-44ad-afc8-102cb132e193" containerName="payload-processing"
Nov 01 05:43:54.606342 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:54.606299    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9118c67d-7bca-44ad-afc8-102cb132e193" containerName="payload-processing"
Nov 01 05:43:54.606342 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:54.606330    5689 cpu_manager.go:414] "containerMap: removing container" podUID="e5664a5e-da43-4b3b-8d52-3789fca3d216" containerName="payload-pre-processing"
Nov 01 05:43:54.606406 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:54.606340    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5664a5e-da43-4b3b-8d52-3789fca3d216" containerName="payload-pre-processing"
Nov 01 05:43:54.606444 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:54.606436    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5664a5e-da43-4b3b-8d52-3789fca3d216" containerName="payload-pre-processing"
Nov 01 05:43:54.606510 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:54.606463    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="9118c67d-7bca-44ad-afc8-102cb132e193" containerName="payload-processing"
Nov 01 05:43:54.610998 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:54.610982    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8815d2-7c78d444b-x7v7h"
Nov 01 05:43:54.668737 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:54.668699    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8815d2-7c78d444b-x7v7h"]
Nov 01 05:43:54.714967 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:54.714936    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-8815d2-serving-cert\""
Nov 01 05:43:54.725543 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:54.725519    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzp5j\" (UniqueName: \"kubernetes.io/projected/512e2680-7d4f-46c1-895a-606f33647f60-kube-api-access-lzp5j\") pod \"maas-api-e2e-api-b-8815d2-7c78d444b-x7v7h\" (UID: \"512e2680-7d4f-46c1-895a-606f33647f60\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8815d2-7c78d444b-x7v7h"
Nov 01 05:43:54.725685 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:54.725663    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/512e2680-7d4f-46c1-895a-606f33647f60-maas-api-tls\") pod \"maas-api-e2e-api-b-8815d2-7c78d444b-x7v7h\" (UID: \"512e2680-7d4f-46c1-895a-606f33647f60\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8815d2-7c78d444b-x7v7h"
Nov 01 05:43:54.725726 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:54.725718    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/512e2680-7d4f-46c1-895a-606f33647f60-metrics-certs\") pod \"maas-api-e2e-api-b-8815d2-7c78d444b-x7v7h\" (UID: \"512e2680-7d4f-46c1-895a-606f33647f60\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8815d2-7c78d444b-x7v7h"
Nov 01 05:43:54.725761 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:54.725742    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/512e2680-7d4f-46c1-895a-606f33647f60-openshift-service-ca\") pod \"maas-api-e2e-api-b-8815d2-7c78d444b-x7v7h\" (UID: \"512e2680-7d4f-46c1-895a-606f33647f60\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8815d2-7c78d444b-x7v7h"
Nov 01 05:43:54.827434 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:54.827401    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/512e2680-7d4f-46c1-895a-606f33647f60-maas-api-tls\") pod \"maas-api-e2e-api-b-8815d2-7c78d444b-x7v7h\" (UID: \"512e2680-7d4f-46c1-895a-606f33647f60\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8815d2-7c78d444b-x7v7h"
Nov 01 05:43:54.827434 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:54.827436    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/512e2680-7d4f-46c1-895a-606f33647f60-metrics-certs\") pod \"maas-api-e2e-api-b-8815d2-7c78d444b-x7v7h\" (UID: \"512e2680-7d4f-46c1-895a-606f33647f60\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8815d2-7c78d444b-x7v7h"
Nov 01 05:43:54.827664 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:54.827489    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/512e2680-7d4f-46c1-895a-606f33647f60-openshift-service-ca\") pod \"maas-api-e2e-api-b-8815d2-7c78d444b-x7v7h\" (UID: \"512e2680-7d4f-46c1-895a-606f33647f60\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8815d2-7c78d444b-x7v7h"
Nov 01 05:43:54.827664 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:43:54.827546    5689 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-8815d2-serving-cert: secret "maas-api-e2e-api-b-8815d2-serving-cert" not found
Nov 01 05:43:54.827664 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:43:54.827619    5689 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/512e2680-7d4f-46c1-895a-606f33647f60-maas-api-tls podName:512e2680-7d4f-46c1-895a-606f33647f60 nodeName:}" failed. No retries permitted until 2026-11-01 05:43:55.32760157 +0000 UTC m=+2413.398398293 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/512e2680-7d4f-46c1-895a-606f33647f60-maas-api-tls") pod "maas-api-e2e-api-b-8815d2-7c78d444b-x7v7h" (UID: "512e2680-7d4f-46c1-895a-606f33647f60") : secret "maas-api-e2e-api-b-8815d2-serving-cert" not found
Nov 01 05:43:54.827664 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:54.827647    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzp5j\" (UniqueName: \"kubernetes.io/projected/512e2680-7d4f-46c1-895a-606f33647f60-kube-api-access-lzp5j\") pod \"maas-api-e2e-api-b-8815d2-7c78d444b-x7v7h\" (UID: \"512e2680-7d4f-46c1-895a-606f33647f60\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8815d2-7c78d444b-x7v7h"
Nov 01 05:43:54.828298 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:54.828274    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/512e2680-7d4f-46c1-895a-606f33647f60-openshift-service-ca\") pod \"maas-api-e2e-api-b-8815d2-7c78d444b-x7v7h\" (UID: \"512e2680-7d4f-46c1-895a-606f33647f60\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8815d2-7c78d444b-x7v7h"
Nov 01 05:43:54.830036 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:54.830021    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/512e2680-7d4f-46c1-895a-606f33647f60-metrics-certs\") pod \"maas-api-e2e-api-b-8815d2-7c78d444b-x7v7h\" (UID: \"512e2680-7d4f-46c1-895a-606f33647f60\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8815d2-7c78d444b-x7v7h"
Nov 01 05:43:54.892097 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:54.892018    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzp5j\" (UniqueName: \"kubernetes.io/projected/512e2680-7d4f-46c1-895a-606f33647f60-kube-api-access-lzp5j\") pod \"maas-api-e2e-api-b-8815d2-7c78d444b-x7v7h\" (UID: \"512e2680-7d4f-46c1-895a-606f33647f60\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8815d2-7c78d444b-x7v7h"
Nov 01 05:43:54.897114 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:54.897052    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:43:55.074716 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:55.074684    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:43:55.337282 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:55.337251    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/512e2680-7d4f-46c1-895a-606f33647f60-maas-api-tls\") pod \"maas-api-e2e-api-b-8815d2-7c78d444b-x7v7h\" (UID: \"512e2680-7d4f-46c1-895a-606f33647f60\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8815d2-7c78d444b-x7v7h"
Nov 01 05:43:55.339643 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:55.339616    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/512e2680-7d4f-46c1-895a-606f33647f60-maas-api-tls\") pod \"maas-api-e2e-api-b-8815d2-7c78d444b-x7v7h\" (UID: \"512e2680-7d4f-46c1-895a-606f33647f60\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8815d2-7c78d444b-x7v7h"
Nov 01 05:43:55.524735 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:55.524703    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8815d2-7c78d444b-x7v7h"
Nov 01 05:43:55.902562 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:55.902432    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8815d2-7c78d444b-x7v7h"]
Nov 01 05:43:55.905614 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:43:55.905582    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod512e2680_7d4f_46c1_895a_606f33647f60.slice/crio-4451465c1077d26b6f47d57ec5f2394ac4ec6c3232c80c012abf3eb614acdebd WatchSource:0}: Error finding container 4451465c1077d26b6f47d57ec5f2394ac4ec6c3232c80c012abf3eb614acdebd: Status 404 returned error can't find the container with id 4451465c1077d26b6f47d57ec5f2394ac4ec6c3232c80c012abf3eb614acdebd
Nov 01 05:43:56.416148 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:56.416113    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8815d2-7c78d444b-x7v7h" event={"ID":"512e2680-7d4f-46c1-895a-606f33647f60","Type":"ContainerStarted","Data":"4451465c1077d26b6f47d57ec5f2394ac4ec6c3232c80c012abf3eb614acdebd"}
Nov 01 05:43:57.423310 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:57.423274    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8815d2-7c78d444b-x7v7h" event={"ID":"512e2680-7d4f-46c1-895a-606f33647f60","Type":"ContainerStarted","Data":"fef338b9a7d5536cc1a6fd906ddbab6320b7de309537df032332abc317061e76"}
Nov 01 05:43:57.423777 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:57.423357    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8815d2-7c78d444b-x7v7h"
Nov 01 05:43:57.541956 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:57.541900    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8815d2-7c78d444b-x7v7h" podStartSLOduration=2.5353857509999997 podStartE2EDuration="3.54188627s" totalImagesPullingTime="1.006500519s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:43:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:43:57.539966841 +0000 UTC m=+2415.610763610" watchObservedRunningTime="2026-11-01 05:43:57.54188627 +0000 UTC m=+2415.612682989"
Nov 01 05:43:59.786193 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:43:59.786163    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:44:00.068430 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:44:00.068351    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:44:03.434420 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:44:03.434394    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8815d2-7c78d444b-x7v7h"
Nov 01 05:44:04.009916 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:44:04.009883    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:44:05.104558 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:44:05.104517    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:44:05.278052 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:44:05.278013    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:44:05.581316 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:44:05.581282    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:44:05.820024 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:44:05.819992    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:44:06.011105 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:44:06.011063    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:44:06.349587 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:44:06.349444    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:44:06.515341 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:44:06.515287    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:44:06.714220 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:44:06.714184    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:44:07.228992 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:44:07.228964    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:44:07.398321 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:44:07.398292    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:44:08.301346 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:44:08.301301    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:44:08.594535 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:44:08.593872    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:44:10.945617 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:44:10.945431    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:44:11.355793 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:44:11.355711    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:44:11.539329 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:44:11.539279    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:44:11.711393 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:44:11.711357    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:44:14.582138 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:44:14.582067    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:44:14.757025 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:44:14.756992    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:44:16.134002 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:44:16.133965    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:44:17.565590 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:44:17.565554    5689 generic.go:356] "Generic (PLEG): container finished" podID="e28aa73d-02c7-4b5f-9d59-fca29e73cc89" containerID="474e0d5e5db9511ac78c298d96bb5f69b8446299a7edcb72b7c426b8cb722819" exitCode=0
Nov 01 05:44:17.566000 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:44:17.565629    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd" event={"ID":"e28aa73d-02c7-4b5f-9d59-fca29e73cc89","Type":"ContainerDied","Data":"474e0d5e5db9511ac78c298d96bb5f69b8446299a7edcb72b7c426b8cb722819"}
Nov 01 05:44:17.566000 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:44:17.565976    5689 scope.go:122] "RemoveContainer" containerID="474e0d5e5db9511ac78c298d96bb5f69b8446299a7edcb72b7c426b8cb722819"
Nov 01 05:44:17.567496 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:44:17.567439    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:44:17.567561 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:44:17.567538    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:44:17.567597 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:44:17.567584    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:44:18.572636 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:44:18.572604    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd" event={"ID":"e28aa73d-02c7-4b5f-9d59-fca29e73cc89","Type":"ContainerStarted","Data":"278da1e1060ea639ba5df64e453f45bda56b2cee3431a33c580e7ce31d77c67f"}
Nov 01 05:44:18.593342 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:44:18.593132    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd"
Nov 01 05:44:18.603162 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:44:18.603120    5689 patch_prober.go:29] interesting pod/e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Nov 01 05:44:18.603361 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:44:18.603190    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd" podUID="e28aa73d-02c7-4b5f-9d59-fca29e73cc89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Nov 01 05:44:18.824860 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:44:18.824775    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:44:19.001484 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:44:19.001030    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:44:19.255115 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:44:19.255079    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:44:19.602876 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:44:19.602770    5689 patch_prober.go:29] interesting pod/e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Nov 01 05:44:19.602876 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:44:19.602844    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd" podUID="e28aa73d-02c7-4b5f-9d59-fca29e73cc89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Nov 01 05:44:20.110365 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:44:20.110262    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:44:20.598921 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:44:20.598883    5689 patch_prober.go:29] interesting pod/e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Nov 01 05:44:20.599228 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:44:20.599193    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd" podUID="e28aa73d-02c7-4b5f-9d59-fca29e73cc89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Nov 01 05:44:20.904373 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:44:20.904345    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:44:21.595831 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:44:21.595795    5689 patch_prober.go:29] interesting pod/e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Nov 01 05:44:21.596099 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:44:21.596074    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd" podUID="e28aa73d-02c7-4b5f-9d59-fca29e73cc89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Nov 01 05:44:22.599829 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:44:22.599794    5689 patch_prober.go:29] interesting pod/e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Nov 01 05:44:22.600229 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:44:22.599866    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd" podUID="e28aa73d-02c7-4b5f-9d59-fca29e73cc89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Nov 01 05:44:22.995100 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:44:22.995069    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd"
Nov 01 05:44:23.693964 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:44:23.693919    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd"
Nov 01 05:44:23.696018 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:44:23.695973    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd"
Nov 01 05:44:25.653573 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:44:25.653527    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:44:25.839479 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:44:25.838595    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:44:26.017970 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:44:26.017886    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:44:29.064764 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:44:29.064728    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:44:30.068638 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:44:30.068604    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:44:30.601730 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:44:30.601698    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:44:32.733260 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:44:32.733227    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:44:32.924505 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:44:32.924439    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:44:33.156724 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:44:33.156697    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:44:35.414252 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:44:35.414211    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:44:40.205092 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:44:40.205061    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:44:45.001979 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:44:45.001945    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:44:45.862859 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:44:45.862826    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:44:46.048373 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:44:46.048305    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:44:46.305602 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:44:46.305568    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:44:49.810432 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:44:49.810387    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:44:54.551593 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:44:54.551564    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:44:59.024289 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:44:59.024250    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:44:59.202754 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:44:59.202723    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:44:59.375369 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:44:59.375287    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:44:59.569783 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:44:59.569728    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:44:59.796403 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:44:59.796364    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:00.076248 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:45:00.076158    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:45:00.211845 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:00.211812    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a48099-29891865-rhlg9"]
Nov 01 05:45:00.216101 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:00.216083    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a48099-29891865-rhlg9"
Nov 01 05:45:00.326747 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:00.326657    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a48099-29891865-rhlg9"]
Nov 01 05:45:00.328867 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:00.328845    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\""
Nov 01 05:45:00.352837 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:00.352804    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8dab737-0a11-43d7-98ce-48de53f062a4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-a48099-29891865-rhlg9\" (UID: \"a8dab737-0a11-43d7-98ce-48de53f062a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a48099-29891865-rhlg9"
Nov 01 05:45:00.352983 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:00.352928    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkxcd\" (UniqueName: \"kubernetes.io/projected/a8dab737-0a11-43d7-98ce-48de53f062a4-kube-api-access-kkxcd\") pod \"maas-api-key-cleanup-e2e-worker-w4-a48099-29891865-rhlg9\" (UID: \"a8dab737-0a11-43d7-98ce-48de53f062a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a48099-29891865-rhlg9"
Nov 01 05:45:00.454096 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:00.454061    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkxcd\" (UniqueName: \"kubernetes.io/projected/a8dab737-0a11-43d7-98ce-48de53f062a4-kube-api-access-kkxcd\") pod \"maas-api-key-cleanup-e2e-worker-w4-a48099-29891865-rhlg9\" (UID: \"a8dab737-0a11-43d7-98ce-48de53f062a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a48099-29891865-rhlg9"
Nov 01 05:45:00.454271 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:00.454256    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8dab737-0a11-43d7-98ce-48de53f062a4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-a48099-29891865-rhlg9\" (UID: \"a8dab737-0a11-43d7-98ce-48de53f062a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a48099-29891865-rhlg9"
Nov 01 05:45:00.454874 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:00.454847    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8dab737-0a11-43d7-98ce-48de53f062a4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-a48099-29891865-rhlg9\" (UID: \"a8dab737-0a11-43d7-98ce-48de53f062a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a48099-29891865-rhlg9"
Nov 01 05:45:00.514419 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:00.514393    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkxcd\" (UniqueName: \"kubernetes.io/projected/a8dab737-0a11-43d7-98ce-48de53f062a4-kube-api-access-kkxcd\") pod \"maas-api-key-cleanup-e2e-worker-w4-a48099-29891865-rhlg9\" (UID: \"a8dab737-0a11-43d7-98ce-48de53f062a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a48099-29891865-rhlg9"
Nov 01 05:45:00.529385 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:00.529362    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a48099-29891865-rhlg9"
Nov 01 05:45:00.903010 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:00.902987    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a48099-29891865-rhlg9"]
Nov 01 05:45:00.906020 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:45:00.905993    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8dab737_0a11_43d7_98ce_48de53f062a4.slice/crio-2a9b3d209d2ea512d03103941d01186983dedd2dd96cf4ba8d13dc75e51089fa WatchSource:0}: Error finding container 2a9b3d209d2ea512d03103941d01186983dedd2dd96cf4ba8d13dc75e51089fa: Status 404 returned error can't find the container with id 2a9b3d209d2ea512d03103941d01186983dedd2dd96cf4ba8d13dc75e51089fa
Nov 01 05:45:01.876773 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:01.876741    5689 generic.go:356] "Generic (PLEG): container finished" podID="a8dab737-0a11-43d7-98ce-48de53f062a4" containerID="c86bb59585910545defeafeeeaa2321c1cf0cbc741318f5cbd3814f25a21abe6" exitCode=60
Nov 01 05:45:01.877137 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:01.876782    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a48099-29891865-rhlg9" event={"ID":"a8dab737-0a11-43d7-98ce-48de53f062a4","Type":"ContainerDied","Data":"c86bb59585910545defeafeeeaa2321c1cf0cbc741318f5cbd3814f25a21abe6"}
Nov 01 05:45:01.877137 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:01.876816    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a48099-29891865-rhlg9" event={"ID":"a8dab737-0a11-43d7-98ce-48de53f062a4","Type":"ContainerStarted","Data":"2a9b3d209d2ea512d03103941d01186983dedd2dd96cf4ba8d13dc75e51089fa"}
Nov 01 05:45:01.877137 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:01.877034    5689 scope.go:122] "RemoveContainer" containerID="c86bb59585910545defeafeeeaa2321c1cf0cbc741318f5cbd3814f25a21abe6"
Nov 01 05:45:02.883788 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:02.883753    5689 generic.go:356] "Generic (PLEG): container finished" podID="a8dab737-0a11-43d7-98ce-48de53f062a4" containerID="973ad50410650b277176a844321fa105cc3e2916ee20edaef0d0310bbd000871" exitCode=60
Nov 01 05:45:02.884231 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:02.883809    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a48099-29891865-rhlg9" event={"ID":"a8dab737-0a11-43d7-98ce-48de53f062a4","Type":"ContainerDied","Data":"973ad50410650b277176a844321fa105cc3e2916ee20edaef0d0310bbd000871"}
Nov 01 05:45:02.884231 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:02.883884    5689 scope.go:122] "RemoveContainer" containerID="c86bb59585910545defeafeeeaa2321c1cf0cbc741318f5cbd3814f25a21abe6"
Nov 01 05:45:02.884231 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:02.884130    5689 scope.go:122] "RemoveContainer" containerID="973ad50410650b277176a844321fa105cc3e2916ee20edaef0d0310bbd000871"
Nov 01 05:45:02.884410 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:45:02.884390    5689 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-a48099-29891865-rhlg9_odh-ai-gateway-infra(a8dab737-0a11-43d7-98ce-48de53f062a4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a48099-29891865-rhlg9" podUID="a8dab737-0a11-43d7-98ce-48de53f062a4"
Nov 01 05:45:03.891533 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:03.891499    5689 scope.go:122] "RemoveContainer" containerID="973ad50410650b277176a844321fa105cc3e2916ee20edaef0d0310bbd000871"
Nov 01 05:45:03.891901 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:45:03.891773    5689 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-a48099-29891865-rhlg9_odh-ai-gateway-infra(a8dab737-0a11-43d7-98ce-48de53f062a4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a48099-29891865-rhlg9" podUID="a8dab737-0a11-43d7-98ce-48de53f062a4"
Nov 01 05:45:04.214181 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:04.214146    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:06.676282 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:06.676238    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:06.858948 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:06.858916    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:07.683788 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:07.683728    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:07.872761 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:07.872724    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:08.899139 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:08.899109    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:45:09.325565 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:09.325489    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:11.864839 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:11.864808    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:12.637472 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:12.637408    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:12.870659 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:12.870590    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:13.824637 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:13.824600    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:14.003990 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:14.003949    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:14.182019 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:14.181986    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:14.365683 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:14.365653    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:16.069036 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:16.069011    5689 scope.go:122] "RemoveContainer" containerID="973ad50410650b277176a844321fa105cc3e2916ee20edaef0d0310bbd000871"
Nov 01 05:45:16.975491 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:16.975441    5689 generic.go:356] "Generic (PLEG): container finished" podID="a8dab737-0a11-43d7-98ce-48de53f062a4" containerID="95caab43af56cabc73db8670d77e89cbca8b7f1cc42e2edd18214c1a7c774835" exitCode=60
Nov 01 05:45:16.975491 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:16.975482    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a48099-29891865-rhlg9" event={"ID":"a8dab737-0a11-43d7-98ce-48de53f062a4","Type":"ContainerDied","Data":"95caab43af56cabc73db8670d77e89cbca8b7f1cc42e2edd18214c1a7c774835"}
Nov 01 05:45:16.975706 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:16.975542    5689 scope.go:122] "RemoveContainer" containerID="973ad50410650b277176a844321fa105cc3e2916ee20edaef0d0310bbd000871"
Nov 01 05:45:16.975915 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:16.975896    5689 scope.go:122] "RemoveContainer" containerID="95caab43af56cabc73db8670d77e89cbca8b7f1cc42e2edd18214c1a7c774835"
Nov 01 05:45:16.976250 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:45:16.976224    5689 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-a48099-29891865-rhlg9_odh-ai-gateway-infra(a8dab737-0a11-43d7-98ce-48de53f062a4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a48099-29891865-rhlg9" podUID="a8dab737-0a11-43d7-98ce-48de53f062a4"
Nov 01 05:45:17.204235 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:17.204203    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:18.165488 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:18.162307    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a48099-29891865-rhlg9"]
Nov 01 05:45:18.405056 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:18.405031    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a48099-29891865-rhlg9"
Nov 01 05:45:18.493382 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:18.493302    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8dab737-0a11-43d7-98ce-48de53f062a4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8dab737-0a11-43d7-98ce-48de53f062a4-maas-api-service-ca\") pod \"a8dab737-0a11-43d7-98ce-48de53f062a4\" (UID: \"a8dab737-0a11-43d7-98ce-48de53f062a4\") "
Nov 01 05:45:18.493382 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:18.493348    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8dab737-0a11-43d7-98ce-48de53f062a4-kube-api-access-kkxcd\" (UniqueName: \"kubernetes.io/projected/a8dab737-0a11-43d7-98ce-48de53f062a4-kube-api-access-kkxcd\") pod \"a8dab737-0a11-43d7-98ce-48de53f062a4\" (UID: \"a8dab737-0a11-43d7-98ce-48de53f062a4\") "
Nov 01 05:45:18.493720 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:18.493697    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8dab737-0a11-43d7-98ce-48de53f062a4-maas-api-service-ca" pod "a8dab737-0a11-43d7-98ce-48de53f062a4" (UID: "a8dab737-0a11-43d7-98ce-48de53f062a4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:45:18.493871 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:18.493855    5689 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8dab737-0a11-43d7-98ce-48de53f062a4-maas-api-service-ca\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:45:18.495571 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:18.495548    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8dab737-0a11-43d7-98ce-48de53f062a4-kube-api-access-kkxcd" pod "a8dab737-0a11-43d7-98ce-48de53f062a4" (UID: "a8dab737-0a11-43d7-98ce-48de53f062a4"). InnerVolumeSpecName "kube-api-access-kkxcd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:45:18.594933 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:18.594890    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkxcd\" (UniqueName: \"kubernetes.io/projected/a8dab737-0a11-43d7-98ce-48de53f062a4-kube-api-access-kkxcd\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:45:18.850829 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:18.850744    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:18.990295 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:18.990270    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a48099-29891865-rhlg9"
Nov 01 05:45:18.990439 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:18.990305    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a48099-29891865-rhlg9" event={"ID":"a8dab737-0a11-43d7-98ce-48de53f062a4","Type":"ContainerDied","Data":"2a9b3d209d2ea512d03103941d01186983dedd2dd96cf4ba8d13dc75e51089fa"}
Nov 01 05:45:18.990439 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:18.990368    5689 scope.go:122] "RemoveContainer" containerID="95caab43af56cabc73db8670d77e89cbca8b7f1cc42e2edd18214c1a7c774835"
Nov 01 05:45:19.055060 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:19.052153    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:19.161716 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:19.161680    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a48099-29891865-rhlg9"]
Nov 01 05:45:19.167491 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:19.167464    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a48099-29891865-rhlg9"]
Nov 01 05:45:19.305528 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:19.305500    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:20.066989 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:20.066960    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8dab737-0a11-43d7-98ce-48de53f062a4" path="/var/lib/kubelet/pods/a8dab737-0a11-43d7-98ce-48de53f062a4/volumes"
Nov 01 05:45:24.533850 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:24.533798    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:24.805547 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:24.805437    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:29.092122 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:29.092081    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:29.281476 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:29.281414    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:30.075606 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:45:30.075566    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:45:31.091093 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:31.091057    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:36.315902 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:36.315870    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:41.239431 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:41.239397    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:41.417062 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:41.417034    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:41.592544 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:41.592441    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:44.040195 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:44.040158    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:44.903674 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:44.903639    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:46.616988 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:46.616959    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:46.939102 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:46.939062    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:45:48.794479 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:48.785380    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:51.596485 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:51.596388    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:53.252890 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:53.252857    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zml98"]
Nov 01 05:45:53.253402 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:53.253387    5689 cpu_manager.go:414] "containerMap: removing container" podUID="a8dab737-0a11-43d7-98ce-48de53f062a4" containerName="cleanup"
Nov 01 05:45:53.253402 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:53.253402    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8dab737-0a11-43d7-98ce-48de53f062a4" containerName="cleanup"
Nov 01 05:45:53.253515 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:53.253420    5689 cpu_manager.go:414] "containerMap: removing container" podUID="a8dab737-0a11-43d7-98ce-48de53f062a4" containerName="cleanup"
Nov 01 05:45:53.253515 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:53.253425    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8dab737-0a11-43d7-98ce-48de53f062a4" containerName="cleanup"
Nov 01 05:45:53.253593 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:53.253562    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8dab737-0a11-43d7-98ce-48de53f062a4" containerName="cleanup"
Nov 01 05:45:53.253593 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:53.253573    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8dab737-0a11-43d7-98ce-48de53f062a4" containerName="cleanup"
Nov 01 05:45:53.253593 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:53.253582    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8dab737-0a11-43d7-98ce-48de53f062a4" containerName="cleanup"
Nov 01 05:45:53.257852 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:53.257836    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zml98"
Nov 01 05:45:53.396069 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:53.396031    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/758d79fd-92ca-46fb-b487-694dc9818b61-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-zml98\" (UID: \"758d79fd-92ca-46fb-b487-694dc9818b61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zml98"
Nov 01 05:45:53.396256 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:53.396088    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/758d79fd-92ca-46fb-b487-694dc9818b61-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-zml98\" (UID: \"758d79fd-92ca-46fb-b487-694dc9818b61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zml98"
Nov 01 05:45:53.396256 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:53.396186    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/758d79fd-92ca-46fb-b487-694dc9818b61-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-zml98\" (UID: \"758d79fd-92ca-46fb-b487-694dc9818b61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zml98"
Nov 01 05:45:53.396386 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:53.396293    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/758d79fd-92ca-46fb-b487-694dc9818b61-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-zml98\" (UID: \"758d79fd-92ca-46fb-b487-694dc9818b61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zml98"
Nov 01 05:45:53.396386 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:53.396315    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/758d79fd-92ca-46fb-b487-694dc9818b61-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-zml98\" (UID: \"758d79fd-92ca-46fb-b487-694dc9818b61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zml98"
Nov 01 05:45:53.396527 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:53.396445    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rfw4\" (UniqueName: \"kubernetes.io/projected/758d79fd-92ca-46fb-b487-694dc9818b61-kube-api-access-4rfw4\") pod \"maas-default-gateway-openshift-default-58d898bd55-zml98\" (UID: \"758d79fd-92ca-46fb-b487-694dc9818b61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zml98"
Nov 01 05:45:53.396527 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:53.396513    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/758d79fd-92ca-46fb-b487-694dc9818b61-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-zml98\" (UID: \"758d79fd-92ca-46fb-b487-694dc9818b61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zml98"
Nov 01 05:45:53.396637 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:53.396569    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/758d79fd-92ca-46fb-b487-694dc9818b61-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-zml98\" (UID: \"758d79fd-92ca-46fb-b487-694dc9818b61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zml98"
Nov 01 05:45:53.396637 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:53.396599    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/758d79fd-92ca-46fb-b487-694dc9818b61-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-zml98\" (UID: \"758d79fd-92ca-46fb-b487-694dc9818b61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zml98"
Nov 01 05:45:53.461076 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:53.461041    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zml98"]
Nov 01 05:45:53.497644 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:53.497606    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/758d79fd-92ca-46fb-b487-694dc9818b61-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-zml98\" (UID: \"758d79fd-92ca-46fb-b487-694dc9818b61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zml98"
Nov 01 05:45:53.497857 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:53.497741    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/758d79fd-92ca-46fb-b487-694dc9818b61-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-zml98\" (UID: \"758d79fd-92ca-46fb-b487-694dc9818b61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zml98"
Nov 01 05:45:53.497857 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:53.497773    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/758d79fd-92ca-46fb-b487-694dc9818b61-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-zml98\" (UID: \"758d79fd-92ca-46fb-b487-694dc9818b61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zml98"
Nov 01 05:45:53.497857 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:53.497841    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rfw4\" (UniqueName: \"kubernetes.io/projected/758d79fd-92ca-46fb-b487-694dc9818b61-kube-api-access-4rfw4\") pod \"maas-default-gateway-openshift-default-58d898bd55-zml98\" (UID: \"758d79fd-92ca-46fb-b487-694dc9818b61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zml98"
Nov 01 05:45:53.498070 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:53.497875    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/758d79fd-92ca-46fb-b487-694dc9818b61-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-zml98\" (UID: \"758d79fd-92ca-46fb-b487-694dc9818b61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zml98"
Nov 01 05:45:53.498070 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:53.497904    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/758d79fd-92ca-46fb-b487-694dc9818b61-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-zml98\" (UID: \"758d79fd-92ca-46fb-b487-694dc9818b61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zml98"
Nov 01 05:45:53.498070 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:53.497935    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/758d79fd-92ca-46fb-b487-694dc9818b61-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-zml98\" (UID: \"758d79fd-92ca-46fb-b487-694dc9818b61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zml98"
Nov 01 05:45:53.498070 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:53.498001    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/758d79fd-92ca-46fb-b487-694dc9818b61-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-zml98\" (UID: \"758d79fd-92ca-46fb-b487-694dc9818b61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zml98"
Nov 01 05:45:53.498070 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:53.498053    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/758d79fd-92ca-46fb-b487-694dc9818b61-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-zml98\" (UID: \"758d79fd-92ca-46fb-b487-694dc9818b61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zml98"
Nov 01 05:45:53.498845 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:53.498814    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/758d79fd-92ca-46fb-b487-694dc9818b61-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-zml98\" (UID: \"758d79fd-92ca-46fb-b487-694dc9818b61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zml98"
Nov 01 05:45:53.499103 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:53.499080    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/758d79fd-92ca-46fb-b487-694dc9818b61-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-zml98\" (UID: \"758d79fd-92ca-46fb-b487-694dc9818b61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zml98"
Nov 01 05:45:53.499340 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:53.499220    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/758d79fd-92ca-46fb-b487-694dc9818b61-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-zml98\" (UID: \"758d79fd-92ca-46fb-b487-694dc9818b61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zml98"
Nov 01 05:45:53.499340 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:53.499300    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/758d79fd-92ca-46fb-b487-694dc9818b61-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-zml98\" (UID: \"758d79fd-92ca-46fb-b487-694dc9818b61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zml98"
Nov 01 05:45:53.499740 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:53.499716    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/758d79fd-92ca-46fb-b487-694dc9818b61-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-zml98\" (UID: \"758d79fd-92ca-46fb-b487-694dc9818b61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zml98"
Nov 01 05:45:53.500374 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:53.500355    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/758d79fd-92ca-46fb-b487-694dc9818b61-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-zml98\" (UID: \"758d79fd-92ca-46fb-b487-694dc9818b61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zml98"
Nov 01 05:45:53.501622 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:53.501603    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/758d79fd-92ca-46fb-b487-694dc9818b61-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-zml98\" (UID: \"758d79fd-92ca-46fb-b487-694dc9818b61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zml98"
Nov 01 05:45:53.558063 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:53.557997    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/758d79fd-92ca-46fb-b487-694dc9818b61-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-zml98\" (UID: \"758d79fd-92ca-46fb-b487-694dc9818b61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zml98"
Nov 01 05:45:53.558342 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:53.558323    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rfw4\" (UniqueName: \"kubernetes.io/projected/758d79fd-92ca-46fb-b487-694dc9818b61-kube-api-access-4rfw4\") pod \"maas-default-gateway-openshift-default-58d898bd55-zml98\" (UID: \"758d79fd-92ca-46fb-b487-694dc9818b61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zml98"
Nov 01 05:45:53.571838 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:53.571815    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zml98"
Nov 01 05:45:54.000402 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:54.000266    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zml98"]
Nov 01 05:45:54.003445 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:45:54.003415    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod758d79fd_92ca_46fb_b487_694dc9818b61.slice/crio-cca0c60df1e519b9417f10a553d6b6386fe1fdcf4aa03ccba15d1dd59cca8ecc WatchSource:0}: Error finding container cca0c60df1e519b9417f10a553d6b6386fe1fdcf4aa03ccba15d1dd59cca8ecc: Status 404 returned error can't find the container with id cca0c60df1e519b9417f10a553d6b6386fe1fdcf4aa03ccba15d1dd59cca8ecc
Nov 01 05:45:54.005846 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:54.005817    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:45:54.005927 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:54.005885    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:45:54.005927 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:54.005921    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:45:54.239506 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:54.239433    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zml98" event={"ID":"758d79fd-92ca-46fb-b487-694dc9818b61","Type":"ContainerStarted","Data":"09a876b78766cd3fd778dbb4a32c11412f8beef5e1b7b9cb2e67eec381071c26"}
Nov 01 05:45:54.239664 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:54.239536    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zml98" event={"ID":"758d79fd-92ca-46fb-b487-694dc9818b61","Type":"ContainerStarted","Data":"cca0c60df1e519b9417f10a553d6b6386fe1fdcf4aa03ccba15d1dd59cca8ecc"}
Nov 01 05:45:54.361539 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:54.361404    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zml98" podStartSLOduration=1.361385635 podStartE2EDuration="1.361385635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:45:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:45:54.35936716 +0000 UTC m=+2532.430163880" watchObservedRunningTime="2026-11-01 05:45:54.361385635 +0000 UTC m=+2532.432182356"
Nov 01 05:45:54.572759 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:54.572702    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zml98"
Nov 01 05:45:55.574028 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:55.573999    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-zml98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Nov 01 05:45:55.574420 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:55.574062    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zml98" podUID="758d79fd-92ca-46fb-b487-694dc9818b61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Nov 01 05:45:55.794675 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:55.794637    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:56.585905 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:56.585865    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-zml98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Nov 01 05:45:56.587683 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:56.587650    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zml98" podUID="758d79fd-92ca-46fb-b487-694dc9818b61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Nov 01 05:45:57.530473 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:57.530429    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:57.585219 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:57.585183    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-zml98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Nov 01 05:45:57.585597 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:57.585551    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zml98" podUID="758d79fd-92ca-46fb-b487-694dc9818b61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Nov 01 05:45:58.572804 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:58.572773    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-zml98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Nov 01 05:45:58.573178 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:58.572828    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zml98" podUID="758d79fd-92ca-46fb-b487-694dc9818b61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Nov 01 05:45:58.938840 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:58.938809    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:59.115119 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:59.115086    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:59.572541 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:59.572507    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-zml98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Nov 01 05:45:59.572707 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:45:59.572558    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zml98" podUID="758d79fd-92ca-46fb-b487-694dc9818b61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Nov 01 05:46:00.072317 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:46:00.072280    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:46:00.439275 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:46:00.439240    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:46:00.572914 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:46:00.572882    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-zml98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Nov 01 05:46:00.573084 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:46:00.572945    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zml98" podUID="758d79fd-92ca-46fb-b487-694dc9818b61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Nov 01 05:46:01.044035 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:46:01.044003    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:46:01.572653 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:46:01.572620    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-zml98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Nov 01 05:46:01.573121 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:46:01.572689    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zml98" podUID="758d79fd-92ca-46fb-b487-694dc9818b61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Nov 01 05:46:01.802570 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:46:01.802517    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:46:02.514486 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:46:02.514431    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:46:02.576547 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:46:02.576515    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zml98"
Nov 01 05:46:02.576939 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:46:02.576792    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zml98"
Nov 01 05:46:02.577581 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:46:02.577563    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zml98"
Nov 01 05:46:07.615803 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:46:07.615768    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:46:07.845482 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:46:07.845421    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:46:08.091850 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:46:08.091823    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:46:08.510089 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:46:08.510059    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8815d2-58457b5f85-t72rh"]
Nov 01 05:46:08.916686 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:46:08.916652    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:46:09.098409 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:46:09.098375    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:46:12.619522 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:46:12.619434    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:46:13.413311 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:46:13.413280    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:46:13.510739 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:46:13.510685    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-8815d2-58457b5f85-t72rh" podUID="6fcd6eef-c926-4e83-a83e-e14e2a6ec6e7" containerName="payload-processing" containerID="cri-o://ffd62110ca7bb0ca73e52ae08d91c57cabb01e291da15e7f78bc1b0efa592ad4" gracePeriod=25
Nov 01 05:46:13.841752 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:46:13.841722    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8815d2-58457b5f85-t72rh"
Nov 01 05:46:13.919098 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:46:13.919073    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:46:14.009908 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:46:14.009834    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fcd6eef-c926-4e83-a83e-e14e2a6ec6e7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fcd6eef-c926-4e83-a83e-e14e2a6ec6e7-plugins-config-volume\") pod \"6fcd6eef-c926-4e83-a83e-e14e2a6ec6e7\" (UID: \"6fcd6eef-c926-4e83-a83e-e14e2a6ec6e7\") "
Nov 01 05:46:14.010076 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:46:14.009922    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fcd6eef-c926-4e83-a83e-e14e2a6ec6e7-kube-api-access-f7j77\" (UniqueName: \"kubernetes.io/projected/6fcd6eef-c926-4e83-a83e-e14e2a6ec6e7-kube-api-access-f7j77\") pod \"6fcd6eef-c926-4e83-a83e-e14e2a6ec6e7\" (UID: \"6fcd6eef-c926-4e83-a83e-e14e2a6ec6e7\") "
Nov 01 05:46:14.010262 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:46:14.010235    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fcd6eef-c926-4e83-a83e-e14e2a6ec6e7-plugins-config-volume" pod "6fcd6eef-c926-4e83-a83e-e14e2a6ec6e7" (UID: "6fcd6eef-c926-4e83-a83e-e14e2a6ec6e7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:46:14.010346 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:46:14.010332    5689 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fcd6eef-c926-4e83-a83e-e14e2a6ec6e7-plugins-config-volume\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:46:14.012133 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:46:14.012112    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fcd6eef-c926-4e83-a83e-e14e2a6ec6e7-kube-api-access-f7j77" pod "6fcd6eef-c926-4e83-a83e-e14e2a6ec6e7" (UID: "6fcd6eef-c926-4e83-a83e-e14e2a6ec6e7"). InnerVolumeSpecName "kube-api-access-f7j77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:46:14.097841 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:46:14.097671    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:46:14.111032 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:46:14.110999    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7j77\" (UniqueName: \"kubernetes.io/projected/6fcd6eef-c926-4e83-a83e-e14e2a6ec6e7-kube-api-access-f7j77\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:46:14.355698 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:46:14.355620    5689 generic.go:356] "Generic (PLEG): container finished" podID="6fcd6eef-c926-4e83-a83e-e14e2a6ec6e7" containerID="ffd62110ca7bb0ca73e52ae08d91c57cabb01e291da15e7f78bc1b0efa592ad4" exitCode=0
Nov 01 05:46:14.355698 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:46:14.355662    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8815d2-58457b5f85-t72rh"
Nov 01 05:46:14.355698 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:46:14.355667    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8815d2-58457b5f85-t72rh" event={"ID":"6fcd6eef-c926-4e83-a83e-e14e2a6ec6e7","Type":"ContainerDied","Data":"ffd62110ca7bb0ca73e52ae08d91c57cabb01e291da15e7f78bc1b0efa592ad4"}
Nov 01 05:46:14.355920 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:46:14.355708    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8815d2-58457b5f85-t72rh" event={"ID":"6fcd6eef-c926-4e83-a83e-e14e2a6ec6e7","Type":"ContainerDied","Data":"ac3d9f6e4e8a8b3243df9a938c8db406f08920f9a8ae9f68ccd62b2f3eea8cde"}
Nov 01 05:46:14.355920 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:46:14.355728    5689 scope.go:122] "RemoveContainer" containerID="ffd62110ca7bb0ca73e52ae08d91c57cabb01e291da15e7f78bc1b0efa592ad4"
Nov 01 05:46:14.365515 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:46:14.365495    5689 scope.go:122] "RemoveContainer" containerID="ffd62110ca7bb0ca73e52ae08d91c57cabb01e291da15e7f78bc1b0efa592ad4"
Nov 01 05:46:14.365789 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:46:14.365764    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffd62110ca7bb0ca73e52ae08d91c57cabb01e291da15e7f78bc1b0efa592ad4\": container with ID starting with ffd62110ca7bb0ca73e52ae08d91c57cabb01e291da15e7f78bc1b0efa592ad4 not found: ID does not exist" containerID="ffd62110ca7bb0ca73e52ae08d91c57cabb01e291da15e7f78bc1b0efa592ad4"
Nov 01 05:46:14.365886 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:46:14.365795    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffd62110ca7bb0ca73e52ae08d91c57cabb01e291da15e7f78bc1b0efa592ad4"} err="failed to get container status \"ffd62110ca7bb0ca73e52ae08d91c57cabb01e291da15e7f78bc1b0efa592ad4\": rpc error: code = NotFound desc = could not find container \"ffd62110ca7bb0ca73e52ae08d91c57cabb01e291da15e7f78bc1b0efa592ad4\": container with ID starting with ffd62110ca7bb0ca73e52ae08d91c57cabb01e291da15e7f78bc1b0efa592ad4 not found: ID does not exist"
Nov 01 05:46:14.518917 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:46:14.518884    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8815d2-58457b5f85-t72rh"]
Nov 01 05:46:14.568379 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:46:14.568353    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8815d2-58457b5f85-t72rh"]
Nov 01 05:46:15.008320 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:46:15.008290    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:46:16.069394 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:46:16.069361    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fcd6eef-c926-4e83-a83e-e14e2a6ec6e7" path="/var/lib/kubelet/pods/6fcd6eef-c926-4e83-a83e-e14e2a6ec6e7/volumes"
Nov 01 05:46:16.446353 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:46:16.446321    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:46:16.620373 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:46:16.620326    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:46:18.002407 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:46:18.002339    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:46:19.146534 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:46:19.146500    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:46:22.699670 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:46:22.699638    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:46:22.874932 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:46:22.874893    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:46:23.265044 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:46:23.265017    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:46:26.168983 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:46:26.168946    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:46:26.348866 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:46:26.348830    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:46:27.601296 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:46:27.601262    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:46:28.952901 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:46:28.952867    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:46:29.154373 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:46:29.154315    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:46:30.071302 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:46:30.071264    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:46:32.039400 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:46:32.039364    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:46:32.422119 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:46:32.422093    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:46:34.422512 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:46:34.422425    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:46:35.637115 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:46:35.637071    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:46:35.819087 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:46:35.819041    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:46:36.223715 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:46:36.223668    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:46:37.442642 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:46:37.442607    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:46:37.780173 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:46:37.780098    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:46:37.989485 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:46:37.989177    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:46:40.093967 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:46:40.093939    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:46:42.207583 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:46:42.207551    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:46:44.331251 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:46:44.331210    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:46:44.506585 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:46:44.506555    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:46:47.112341 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:46:47.112314    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:46:47.676433 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:46:47.676407    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:46:48.258051 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:46:48.257972    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:46:48.582720 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:46:48.582643    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:46:52.025159 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:46:52.025122    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:46:52.209289 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:46:52.209258    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:46:52.391288 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:46:52.391206    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:46:57.398863 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:46:57.398829    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:46:59.262049 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:46:59.262015    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:46:59.448477 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:46:59.445232    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:46:59.632013 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:46:59.631928    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:46:59.958028 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:46:59.957983    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:47:00.069676 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:47:00.069644    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:47:00.138805 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:00.138781    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:47:02.692581 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:02.692551    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:47:04.868681 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:04.868625    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:47:07.509801 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:07.509745    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:47:08.456178 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:08.456139    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:47:11.633323 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:11.633282    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8815d2-7c78d444b-x7v7h"]
Nov 01 05:47:11.633855 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:11.633568    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8815d2-7c78d444b-x7v7h" podUID="512e2680-7d4f-46c1-895a-606f33647f60" containerName="maas-api" containerID="cri-o://fef338b9a7d5536cc1a6fd906ddbab6320b7de309537df032332abc317061e76" gracePeriod=30
Nov 01 05:47:11.759609 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:11.759582    5689 generic.go:356] "Generic (PLEG): container finished" podID="512e2680-7d4f-46c1-895a-606f33647f60" containerID="fef338b9a7d5536cc1a6fd906ddbab6320b7de309537df032332abc317061e76" exitCode=0
Nov 01 05:47:11.759724 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:11.759632    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8815d2-7c78d444b-x7v7h" event={"ID":"512e2680-7d4f-46c1-895a-606f33647f60","Type":"ContainerDied","Data":"fef338b9a7d5536cc1a6fd906ddbab6320b7de309537df032332abc317061e76"}
Nov 01 05:47:11.764737 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:11.764715    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:47:11.943111 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:11.943079    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8815d2-7c78d444b-x7v7h"
Nov 01 05:47:11.993876 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:11.993845    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/512e2680-7d4f-46c1-895a-606f33647f60-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/512e2680-7d4f-46c1-895a-606f33647f60-maas-api-tls\") pod \"512e2680-7d4f-46c1-895a-606f33647f60\" (UID: \"512e2680-7d4f-46c1-895a-606f33647f60\") "
Nov 01 05:47:11.994113 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:11.994096    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/512e2680-7d4f-46c1-895a-606f33647f60-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/512e2680-7d4f-46c1-895a-606f33647f60-openshift-service-ca\") pod \"512e2680-7d4f-46c1-895a-606f33647f60\" (UID: \"512e2680-7d4f-46c1-895a-606f33647f60\") "
Nov 01 05:47:11.994182 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:11.994168    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/512e2680-7d4f-46c1-895a-606f33647f60-kube-api-access-lzp5j\" (UniqueName: \"kubernetes.io/projected/512e2680-7d4f-46c1-895a-606f33647f60-kube-api-access-lzp5j\") pod \"512e2680-7d4f-46c1-895a-606f33647f60\" (UID: \"512e2680-7d4f-46c1-895a-606f33647f60\") "
Nov 01 05:47:11.994224 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:11.994202    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/512e2680-7d4f-46c1-895a-606f33647f60-metrics-certs\" (UniqueName: \"kubernetes.io/secret/512e2680-7d4f-46c1-895a-606f33647f60-metrics-certs\") pod \"512e2680-7d4f-46c1-895a-606f33647f60\" (UID: \"512e2680-7d4f-46c1-895a-606f33647f60\") "
Nov 01 05:47:11.994490 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:11.994443    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/512e2680-7d4f-46c1-895a-606f33647f60-openshift-service-ca" pod "512e2680-7d4f-46c1-895a-606f33647f60" (UID: "512e2680-7d4f-46c1-895a-606f33647f60"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:47:11.994638 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:11.994613    5689 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/512e2680-7d4f-46c1-895a-606f33647f60-openshift-service-ca\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:47:11.996208 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:11.996180    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/512e2680-7d4f-46c1-895a-606f33647f60-maas-api-tls" pod "512e2680-7d4f-46c1-895a-606f33647f60" (UID: "512e2680-7d4f-46c1-895a-606f33647f60"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 05:47:11.996392 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:11.996372    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/512e2680-7d4f-46c1-895a-606f33647f60-metrics-certs" pod "512e2680-7d4f-46c1-895a-606f33647f60" (UID: "512e2680-7d4f-46c1-895a-606f33647f60"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 05:47:11.996722 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:11.996703    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/512e2680-7d4f-46c1-895a-606f33647f60-kube-api-access-lzp5j" pod "512e2680-7d4f-46c1-895a-606f33647f60" (UID: "512e2680-7d4f-46c1-895a-606f33647f60"). InnerVolumeSpecName "kube-api-access-lzp5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:47:12.095856 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:12.095824    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzp5j\" (UniqueName: \"kubernetes.io/projected/512e2680-7d4f-46c1-895a-606f33647f60-kube-api-access-lzp5j\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:47:12.095856 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:12.095851    5689 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/512e2680-7d4f-46c1-895a-606f33647f60-metrics-certs\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:47:12.096074 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:12.095882    5689 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/512e2680-7d4f-46c1-895a-606f33647f60-maas-api-tls\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:47:12.766670 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:12.766638    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8815d2-7c78d444b-x7v7h"
Nov 01 05:47:12.767179 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:12.766639    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8815d2-7c78d444b-x7v7h" event={"ID":"512e2680-7d4f-46c1-895a-606f33647f60","Type":"ContainerDied","Data":"4451465c1077d26b6f47d57ec5f2394ac4ec6c3232c80c012abf3eb614acdebd"}
Nov 01 05:47:12.767179 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:12.766790    5689 scope.go:122] "RemoveContainer" containerID="fef338b9a7d5536cc1a6fd906ddbab6320b7de309537df032332abc317061e76"
Nov 01 05:47:12.930247 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:12.930212    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8815d2-7c78d444b-x7v7h"]
Nov 01 05:47:12.979903 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:12.979878    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8815d2-7c78d444b-x7v7h"]
Nov 01 05:47:13.199548 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:13.199477    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:47:13.374062 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:13.374031    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:47:13.548992 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:13.548873    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:47:14.067602 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:14.067570    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="512e2680-7d4f-46c1-895a-606f33647f60" path="/var/lib/kubelet/pods/512e2680-7d4f-46c1-895a-606f33647f60/volumes"
Nov 01 05:47:14.762196 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:14.762152    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:47:15.188871 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:15.188839    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd"]
Nov 01 05:47:15.189329 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:15.189098    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd" podUID="e28aa73d-02c7-4b5f-9d59-fca29e73cc89" containerName="istio-proxy" containerID="cri-o://278da1e1060ea639ba5df64e453f45bda56b2cee3431a33c580e7ce31d77c67f" gracePeriod=30
Nov 01 05:47:15.217999 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:15.217976    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:47:18.612847 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:18.612796    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:47:20.461198 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:20.461136    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd"
Nov 01 05:47:20.520880 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:20.520854    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e28aa73d-02c7-4b5f-9d59-fca29e73cc89-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e28aa73d-02c7-4b5f-9d59-fca29e73cc89-istio-podinfo\") pod \"e28aa73d-02c7-4b5f-9d59-fca29e73cc89\" (UID: \"e28aa73d-02c7-4b5f-9d59-fca29e73cc89\") "
Nov 01 05:47:20.521051 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:20.520909    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e28aa73d-02c7-4b5f-9d59-fca29e73cc89-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e28aa73d-02c7-4b5f-9d59-fca29e73cc89-istiod-ca-cert\") pod \"e28aa73d-02c7-4b5f-9d59-fca29e73cc89\" (UID: \"e28aa73d-02c7-4b5f-9d59-fca29e73cc89\") "
Nov 01 05:47:20.521051 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:20.521034    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e28aa73d-02c7-4b5f-9d59-fca29e73cc89-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e28aa73d-02c7-4b5f-9d59-fca29e73cc89-workload-certs\") pod \"e28aa73d-02c7-4b5f-9d59-fca29e73cc89\" (UID: \"e28aa73d-02c7-4b5f-9d59-fca29e73cc89\") "
Nov 01 05:47:20.521175 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:20.521158    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e28aa73d-02c7-4b5f-9d59-fca29e73cc89-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e28aa73d-02c7-4b5f-9d59-fca29e73cc89-credential-socket\") pod \"e28aa73d-02c7-4b5f-9d59-fca29e73cc89\" (UID: \"e28aa73d-02c7-4b5f-9d59-fca29e73cc89\") "
Nov 01 05:47:20.521232 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:20.521192    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e28aa73d-02c7-4b5f-9d59-fca29e73cc89-istio-token\" (UniqueName: \"kubernetes.io/projected/e28aa73d-02c7-4b5f-9d59-fca29e73cc89-istio-token\") pod \"e28aa73d-02c7-4b5f-9d59-fca29e73cc89\" (UID: \"e28aa73d-02c7-4b5f-9d59-fca29e73cc89\") "
Nov 01 05:47:20.521285 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:20.521237    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e28aa73d-02c7-4b5f-9d59-fca29e73cc89-kube-api-access-b6xwv\" (UniqueName: \"kubernetes.io/projected/e28aa73d-02c7-4b5f-9d59-fca29e73cc89-kube-api-access-b6xwv\") pod \"e28aa73d-02c7-4b5f-9d59-fca29e73cc89\" (UID: \"e28aa73d-02c7-4b5f-9d59-fca29e73cc89\") "
Nov 01 05:47:20.521285 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:20.521254    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e28aa73d-02c7-4b5f-9d59-fca29e73cc89-workload-certs" pod "e28aa73d-02c7-4b5f-9d59-fca29e73cc89" (UID: "e28aa73d-02c7-4b5f-9d59-fca29e73cc89"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:47:20.521285 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:20.521266    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e28aa73d-02c7-4b5f-9d59-fca29e73cc89-istiod-ca-cert" pod "e28aa73d-02c7-4b5f-9d59-fca29e73cc89" (UID: "e28aa73d-02c7-4b5f-9d59-fca29e73cc89"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:47:20.521446 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:20.521356    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e28aa73d-02c7-4b5f-9d59-fca29e73cc89-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e28aa73d-02c7-4b5f-9d59-fca29e73cc89-workload-socket\") pod \"e28aa73d-02c7-4b5f-9d59-fca29e73cc89\" (UID: \"e28aa73d-02c7-4b5f-9d59-fca29e73cc89\") "
Nov 01 05:47:20.521446 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:20.521386    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e28aa73d-02c7-4b5f-9d59-fca29e73cc89-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e28aa73d-02c7-4b5f-9d59-fca29e73cc89-istio-data\") pod \"e28aa73d-02c7-4b5f-9d59-fca29e73cc89\" (UID: \"e28aa73d-02c7-4b5f-9d59-fca29e73cc89\") "
Nov 01 05:47:20.521446 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:20.521392    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e28aa73d-02c7-4b5f-9d59-fca29e73cc89-credential-socket" pod "e28aa73d-02c7-4b5f-9d59-fca29e73cc89" (UID: "e28aa73d-02c7-4b5f-9d59-fca29e73cc89"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:47:20.521446 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:20.521425    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e28aa73d-02c7-4b5f-9d59-fca29e73cc89-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e28aa73d-02c7-4b5f-9d59-fca29e73cc89-istio-envoy\") pod \"e28aa73d-02c7-4b5f-9d59-fca29e73cc89\" (UID: \"e28aa73d-02c7-4b5f-9d59-fca29e73cc89\") "
Nov 01 05:47:20.521895 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:20.521844    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e28aa73d-02c7-4b5f-9d59-fca29e73cc89-istio-data" pod "e28aa73d-02c7-4b5f-9d59-fca29e73cc89" (UID: "e28aa73d-02c7-4b5f-9d59-fca29e73cc89"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:47:20.522100 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:20.522016    5689 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e28aa73d-02c7-4b5f-9d59-fca29e73cc89-workload-certs\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:47:20.522100 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:20.522042    5689 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e28aa73d-02c7-4b5f-9d59-fca29e73cc89-credential-socket\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:47:20.522100 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:20.522058    5689 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e28aa73d-02c7-4b5f-9d59-fca29e73cc89-istio-data\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:47:20.522100 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:20.522072    5689 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e28aa73d-02c7-4b5f-9d59-fca29e73cc89-istiod-ca-cert\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:47:20.522423 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:20.522393    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e28aa73d-02c7-4b5f-9d59-fca29e73cc89-workload-socket" pod "e28aa73d-02c7-4b5f-9d59-fca29e73cc89" (UID: "e28aa73d-02c7-4b5f-9d59-fca29e73cc89"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:47:20.523333 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:20.523310    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e28aa73d-02c7-4b5f-9d59-fca29e73cc89-istio-podinfo" pod "e28aa73d-02c7-4b5f-9d59-fca29e73cc89" (UID: "e28aa73d-02c7-4b5f-9d59-fca29e73cc89"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 05:47:20.523519 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:20.523499    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e28aa73d-02c7-4b5f-9d59-fca29e73cc89-kube-api-access-b6xwv" pod "e28aa73d-02c7-4b5f-9d59-fca29e73cc89" (UID: "e28aa73d-02c7-4b5f-9d59-fca29e73cc89"). InnerVolumeSpecName "kube-api-access-b6xwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:47:20.523639 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:20.523620    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e28aa73d-02c7-4b5f-9d59-fca29e73cc89-istio-token" pod "e28aa73d-02c7-4b5f-9d59-fca29e73cc89" (UID: "e28aa73d-02c7-4b5f-9d59-fca29e73cc89"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:47:20.523686 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:20.523639    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e28aa73d-02c7-4b5f-9d59-fca29e73cc89-istio-envoy" pod "e28aa73d-02c7-4b5f-9d59-fca29e73cc89" (UID: "e28aa73d-02c7-4b5f-9d59-fca29e73cc89"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:47:20.554117 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:20.554095    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:47:20.623203 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:20.623176    5689 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e28aa73d-02c7-4b5f-9d59-fca29e73cc89-istio-podinfo\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:47:20.623346 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:20.623206    5689 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e28aa73d-02c7-4b5f-9d59-fca29e73cc89-istio-token\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:47:20.623346 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:20.623222    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6xwv\" (UniqueName: \"kubernetes.io/projected/e28aa73d-02c7-4b5f-9d59-fca29e73cc89-kube-api-access-b6xwv\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:47:20.623346 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:20.623236    5689 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e28aa73d-02c7-4b5f-9d59-fca29e73cc89-workload-socket\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:47:20.623346 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:20.623252    5689 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e28aa73d-02c7-4b5f-9d59-fca29e73cc89-istio-envoy\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:47:20.840800 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:20.840718    5689 generic.go:356] "Generic (PLEG): container finished" podID="e28aa73d-02c7-4b5f-9d59-fca29e73cc89" containerID="278da1e1060ea639ba5df64e453f45bda56b2cee3431a33c580e7ce31d77c67f" exitCode=0
Nov 01 05:47:20.840800 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:20.840771    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd"
Nov 01 05:47:20.841023 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:20.840805    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd" event={"ID":"e28aa73d-02c7-4b5f-9d59-fca29e73cc89","Type":"ContainerDied","Data":"278da1e1060ea639ba5df64e453f45bda56b2cee3431a33c580e7ce31d77c67f"}
Nov 01 05:47:20.841023 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:20.840864    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd" event={"ID":"e28aa73d-02c7-4b5f-9d59-fca29e73cc89","Type":"ContainerDied","Data":"724bc27a35fa490305b50917cd369b66b5ffc06dd8300c5181e433fae3492d30"}
Nov 01 05:47:20.841023 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:20.840890    5689 scope.go:122] "RemoveContainer" containerID="278da1e1060ea639ba5df64e453f45bda56b2cee3431a33c580e7ce31d77c67f"
Nov 01 05:47:20.851393 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:20.851374    5689 scope.go:122] "RemoveContainer" containerID="474e0d5e5db9511ac78c298d96bb5f69b8446299a7edcb72b7c426b8cb722819"
Nov 01 05:47:20.859802 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:20.859784    5689 scope.go:122] "RemoveContainer" containerID="278da1e1060ea639ba5df64e453f45bda56b2cee3431a33c580e7ce31d77c67f"
Nov 01 05:47:20.860060 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:47:20.860042    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"278da1e1060ea639ba5df64e453f45bda56b2cee3431a33c580e7ce31d77c67f\": container with ID starting with 278da1e1060ea639ba5df64e453f45bda56b2cee3431a33c580e7ce31d77c67f not found: ID does not exist" containerID="278da1e1060ea639ba5df64e453f45bda56b2cee3431a33c580e7ce31d77c67f"
Nov 01 05:47:20.860105 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:20.860069    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"278da1e1060ea639ba5df64e453f45bda56b2cee3431a33c580e7ce31d77c67f"} err="failed to get container status \"278da1e1060ea639ba5df64e453f45bda56b2cee3431a33c580e7ce31d77c67f\": rpc error: code = NotFound desc = could not find container \"278da1e1060ea639ba5df64e453f45bda56b2cee3431a33c580e7ce31d77c67f\": container with ID starting with 278da1e1060ea639ba5df64e453f45bda56b2cee3431a33c580e7ce31d77c67f not found: ID does not exist"
Nov 01 05:47:20.860105 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:20.860085    5689 scope.go:122] "RemoveContainer" containerID="474e0d5e5db9511ac78c298d96bb5f69b8446299a7edcb72b7c426b8cb722819"
Nov 01 05:47:20.860314 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:47:20.860296    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"474e0d5e5db9511ac78c298d96bb5f69b8446299a7edcb72b7c426b8cb722819\": container with ID starting with 474e0d5e5db9511ac78c298d96bb5f69b8446299a7edcb72b7c426b8cb722819 not found: ID does not exist" containerID="474e0d5e5db9511ac78c298d96bb5f69b8446299a7edcb72b7c426b8cb722819"
Nov 01 05:47:20.860359 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:20.860321    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"474e0d5e5db9511ac78c298d96bb5f69b8446299a7edcb72b7c426b8cb722819"} err="failed to get container status \"474e0d5e5db9511ac78c298d96bb5f69b8446299a7edcb72b7c426b8cb722819\": rpc error: code = NotFound desc = could not find container \"474e0d5e5db9511ac78c298d96bb5f69b8446299a7edcb72b7c426b8cb722819\": container with ID starting with 474e0d5e5db9511ac78c298d96bb5f69b8446299a7edcb72b7c426b8cb722819 not found: ID does not exist"
Nov 01 05:47:21.006132 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:21.006102    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd"]
Nov 01 05:47:21.055644 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:21.055611    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-8815d2-openshift-default-68fcb7d459-wqcrd"]
Nov 01 05:47:22.067625 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:22.067592    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e28aa73d-02c7-4b5f-9d59-fca29e73cc89" path="/var/lib/kubelet/pods/e28aa73d-02c7-4b5f-9d59-fca29e73cc89/volumes"
Nov 01 05:47:22.830386 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:22.830355    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:47:24.399715 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:24.399681    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:47:24.607086 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:24.607048    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:47:24.781395 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:24.781302    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:47:24.987101 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:24.987056    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:47:26.818555 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:26.818526    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:47:26.997162 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:26.997135    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:47:27.175134 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:27.175106    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:47:27.508498 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:27.508405    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:47:27.696695 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:27.696657    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:47:28.904802 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:28.904773    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:47:29.765721 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:29.765685    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx"]
Nov 01 05:47:29.766291 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:29.766272    5689 cpu_manager.go:414] "containerMap: removing container" podUID="6fcd6eef-c926-4e83-a83e-e14e2a6ec6e7" containerName="payload-processing"
Nov 01 05:47:29.766414 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:29.766293    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fcd6eef-c926-4e83-a83e-e14e2a6ec6e7" containerName="payload-processing"
Nov 01 05:47:29.766414 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:29.766326    5689 cpu_manager.go:414] "containerMap: removing container" podUID="a8dab737-0a11-43d7-98ce-48de53f062a4" containerName="cleanup"
Nov 01 05:47:29.766414 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:29.766334    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8dab737-0a11-43d7-98ce-48de53f062a4" containerName="cleanup"
Nov 01 05:47:29.766414 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:29.766348    5689 cpu_manager.go:414] "containerMap: removing container" podUID="512e2680-7d4f-46c1-895a-606f33647f60" containerName="maas-api"
Nov 01 05:47:29.766414 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:29.766355    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="512e2680-7d4f-46c1-895a-606f33647f60" containerName="maas-api"
Nov 01 05:47:29.766686 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:29.766418    5689 cpu_manager.go:414] "containerMap: removing container" podUID="e28aa73d-02c7-4b5f-9d59-fca29e73cc89" containerName="istio-proxy"
Nov 01 05:47:29.766686 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:29.766427    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e28aa73d-02c7-4b5f-9d59-fca29e73cc89" containerName="istio-proxy"
Nov 01 05:47:29.766686 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:29.766437    5689 cpu_manager.go:414] "containerMap: removing container" podUID="e28aa73d-02c7-4b5f-9d59-fca29e73cc89" containerName="istio-proxy"
Nov 01 05:47:29.766686 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:29.766445    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e28aa73d-02c7-4b5f-9d59-fca29e73cc89" containerName="istio-proxy"
Nov 01 05:47:29.766686 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:29.766611    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="e28aa73d-02c7-4b5f-9d59-fca29e73cc89" containerName="istio-proxy"
Nov 01 05:47:29.766686 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:29.766625    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="e28aa73d-02c7-4b5f-9d59-fca29e73cc89" containerName="istio-proxy"
Nov 01 05:47:29.766686 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:29.766639    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fcd6eef-c926-4e83-a83e-e14e2a6ec6e7" containerName="payload-processing"
Nov 01 05:47:29.766686 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:29.766650    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="512e2680-7d4f-46c1-895a-606f33647f60" containerName="maas-api"
Nov 01 05:47:29.771519 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:29.771500    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx"
Nov 01 05:47:29.816882 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:29.816850    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:47:29.830972 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:29.830942    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d69cb72-f4e4-4ef1-9ceb-210cbd671a54-istio-token\") pod \"e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx\" (UID: \"9d69cb72-f4e4-4ef1-9ceb-210cbd671a54\") " pod="openshift-ingress/e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx"
Nov 01 05:47:29.831126 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:29.830991    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d69cb72-f4e4-4ef1-9ceb-210cbd671a54-workload-certs\") pod \"e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx\" (UID: \"9d69cb72-f4e4-4ef1-9ceb-210cbd671a54\") " pod="openshift-ingress/e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx"
Nov 01 05:47:29.831126 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:29.831117    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d69cb72-f4e4-4ef1-9ceb-210cbd671a54-istio-podinfo\") pod \"e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx\" (UID: \"9d69cb72-f4e4-4ef1-9ceb-210cbd671a54\") " pod="openshift-ingress/e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx"
Nov 01 05:47:29.831220 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:29.831136    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgs5b\" (UniqueName: \"kubernetes.io/projected/9d69cb72-f4e4-4ef1-9ceb-210cbd671a54-kube-api-access-lgs5b\") pod \"e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx\" (UID: \"9d69cb72-f4e4-4ef1-9ceb-210cbd671a54\") " pod="openshift-ingress/e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx"
Nov 01 05:47:29.831220 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:29.831155    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d69cb72-f4e4-4ef1-9ceb-210cbd671a54-credential-socket\") pod \"e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx\" (UID: \"9d69cb72-f4e4-4ef1-9ceb-210cbd671a54\") " pod="openshift-ingress/e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx"
Nov 01 05:47:29.831220 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:29.831174    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d69cb72-f4e4-4ef1-9ceb-210cbd671a54-istio-data\") pod \"e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx\" (UID: \"9d69cb72-f4e4-4ef1-9ceb-210cbd671a54\") " pod="openshift-ingress/e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx"
Nov 01 05:47:29.831419 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:29.831398    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d69cb72-f4e4-4ef1-9ceb-210cbd671a54-workload-socket\") pod \"e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx\" (UID: \"9d69cb72-f4e4-4ef1-9ceb-210cbd671a54\") " pod="openshift-ingress/e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx"
Nov 01 05:47:29.831497 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:29.831466    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d69cb72-f4e4-4ef1-9ceb-210cbd671a54-istiod-ca-cert\") pod \"e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx\" (UID: \"9d69cb72-f4e4-4ef1-9ceb-210cbd671a54\") " pod="openshift-ingress/e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx"
Nov 01 05:47:29.831541 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:29.831524    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d69cb72-f4e4-4ef1-9ceb-210cbd671a54-istio-envoy\") pod \"e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx\" (UID: \"9d69cb72-f4e4-4ef1-9ceb-210cbd671a54\") " pod="openshift-ingress/e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx"
Nov 01 05:47:29.840421 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:29.840394    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx"]
Nov 01 05:47:29.895297 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:29.895261    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-c73135-openshift-default-dockercfg-jszmt\""
Nov 01 05:47:29.945110 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:29.945072    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d69cb72-f4e4-4ef1-9ceb-210cbd671a54-istio-envoy\") pod \"e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx\" (UID: \"9d69cb72-f4e4-4ef1-9ceb-210cbd671a54\") " pod="openshift-ingress/e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx"
Nov 01 05:47:29.957484 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:29.947124    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d69cb72-f4e4-4ef1-9ceb-210cbd671a54-istio-token\") pod \"e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx\" (UID: \"9d69cb72-f4e4-4ef1-9ceb-210cbd671a54\") " pod="openshift-ingress/e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx"
Nov 01 05:47:29.976483 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:29.976030    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d69cb72-f4e4-4ef1-9ceb-210cbd671a54-workload-certs\") pod \"e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx\" (UID: \"9d69cb72-f4e4-4ef1-9ceb-210cbd671a54\") " pod="openshift-ingress/e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx"
Nov 01 05:47:29.976483 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:29.976136    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d69cb72-f4e4-4ef1-9ceb-210cbd671a54-istio-podinfo\") pod \"e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx\" (UID: \"9d69cb72-f4e4-4ef1-9ceb-210cbd671a54\") " pod="openshift-ingress/e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx"
Nov 01 05:47:29.976483 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:29.976161    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgs5b\" (UniqueName: \"kubernetes.io/projected/9d69cb72-f4e4-4ef1-9ceb-210cbd671a54-kube-api-access-lgs5b\") pod \"e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx\" (UID: \"9d69cb72-f4e4-4ef1-9ceb-210cbd671a54\") " pod="openshift-ingress/e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx"
Nov 01 05:47:29.976483 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:29.976193    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d69cb72-f4e4-4ef1-9ceb-210cbd671a54-credential-socket\") pod \"e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx\" (UID: \"9d69cb72-f4e4-4ef1-9ceb-210cbd671a54\") " pod="openshift-ingress/e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx"
Nov 01 05:47:29.976483 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:29.976218    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d69cb72-f4e4-4ef1-9ceb-210cbd671a54-istio-data\") pod \"e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx\" (UID: \"9d69cb72-f4e4-4ef1-9ceb-210cbd671a54\") " pod="openshift-ingress/e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx"
Nov 01 05:47:29.976483 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:29.976362    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d69cb72-f4e4-4ef1-9ceb-210cbd671a54-workload-socket\") pod \"e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx\" (UID: \"9d69cb72-f4e4-4ef1-9ceb-210cbd671a54\") " pod="openshift-ingress/e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx"
Nov 01 05:47:29.976483 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:29.976409    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d69cb72-f4e4-4ef1-9ceb-210cbd671a54-istiod-ca-cert\") pod \"e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx\" (UID: \"9d69cb72-f4e4-4ef1-9ceb-210cbd671a54\") " pod="openshift-ingress/e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx"
Nov 01 05:47:29.977822 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:29.977178    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d69cb72-f4e4-4ef1-9ceb-210cbd671a54-istiod-ca-cert\") pod \"e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx\" (UID: \"9d69cb72-f4e4-4ef1-9ceb-210cbd671a54\") " pod="openshift-ingress/e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx"
Nov 01 05:47:29.977822 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:29.975099    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d69cb72-f4e4-4ef1-9ceb-210cbd671a54-istio-envoy\") pod \"e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx\" (UID: \"9d69cb72-f4e4-4ef1-9ceb-210cbd671a54\") " pod="openshift-ingress/e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx"
Nov 01 05:47:29.977822 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:29.977434    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d69cb72-f4e4-4ef1-9ceb-210cbd671a54-workload-certs\") pod \"e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx\" (UID: \"9d69cb72-f4e4-4ef1-9ceb-210cbd671a54\") " pod="openshift-ingress/e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx"
Nov 01 05:47:29.977822 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:29.977811    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d69cb72-f4e4-4ef1-9ceb-210cbd671a54-credential-socket\") pod \"e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx\" (UID: \"9d69cb72-f4e4-4ef1-9ceb-210cbd671a54\") " pod="openshift-ingress/e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx"
Nov 01 05:47:29.978894 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:29.978343    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d69cb72-f4e4-4ef1-9ceb-210cbd671a54-istio-data\") pod \"e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx\" (UID: \"9d69cb72-f4e4-4ef1-9ceb-210cbd671a54\") " pod="openshift-ingress/e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx"
Nov 01 05:47:29.978894 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:29.978624    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d69cb72-f4e4-4ef1-9ceb-210cbd671a54-workload-socket\") pod \"e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx\" (UID: \"9d69cb72-f4e4-4ef1-9ceb-210cbd671a54\") " pod="openshift-ingress/e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx"
Nov 01 05:47:29.981873 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:29.981848    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d69cb72-f4e4-4ef1-9ceb-210cbd671a54-istio-podinfo\") pod \"e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx\" (UID: \"9d69cb72-f4e4-4ef1-9ceb-210cbd671a54\") " pod="openshift-ingress/e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx"
Nov 01 05:47:29.993377 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:29.993354    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:47:30.019373 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:30.019295    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d69cb72-f4e4-4ef1-9ceb-210cbd671a54-istio-token\") pod \"e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx\" (UID: \"9d69cb72-f4e4-4ef1-9ceb-210cbd671a54\") " pod="openshift-ingress/e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx"
Nov 01 05:47:30.037950 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:30.037921    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgs5b\" (UniqueName: \"kubernetes.io/projected/9d69cb72-f4e4-4ef1-9ceb-210cbd671a54-kube-api-access-lgs5b\") pod \"e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx\" (UID: \"9d69cb72-f4e4-4ef1-9ceb-210cbd671a54\") " pod="openshift-ingress/e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx"
Nov 01 05:47:30.069711 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:47:30.069681    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:47:30.085090 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:30.085067    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx"
Nov 01 05:47:30.440308 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:30.440277    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:47:30.518395 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:30.518356    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx"]
Nov 01 05:47:30.521949 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:47:30.521925    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d69cb72_f4e4_4ef1_9ceb_210cbd671a54.slice/crio-2a448cfa95988a97e98f7b7a57dfef345897e7f7ab336e0a0794e11fbe9e3d6c WatchSource:0}: Error finding container 2a448cfa95988a97e98f7b7a57dfef345897e7f7ab336e0a0794e11fbe9e3d6c: Status 404 returned error can't find the container with id 2a448cfa95988a97e98f7b7a57dfef345897e7f7ab336e0a0794e11fbe9e3d6c
Nov 01 05:47:30.524406 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:30.524376    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:47:30.524497 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:30.524438    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:47:30.524542 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:30.524504    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:47:30.896824 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:30.896732    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx" event={"ID":"9d69cb72-f4e4-4ef1-9ceb-210cbd671a54","Type":"ContainerStarted","Data":"db8bd7e6c8fc393881c66a58e1d3a50e711cf8f9542a92a6752657aac397097b"}
Nov 01 05:47:30.896824 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:30.896785    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx" event={"ID":"9d69cb72-f4e4-4ef1-9ceb-210cbd671a54","Type":"ContainerStarted","Data":"2a448cfa95988a97e98f7b7a57dfef345897e7f7ab336e0a0794e11fbe9e3d6c"}
Nov 01 05:47:31.018408 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:31.018355    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx" podStartSLOduration=2.018341602 podStartE2EDuration="2.018341602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:47:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:47:31.017299539 +0000 UTC m=+2629.088096296" watchObservedRunningTime="2026-11-01 05:47:31.018341602 +0000 UTC m=+2629.089138326"
Nov 01 05:47:31.086213 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:31.086182    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx"
Nov 01 05:47:31.092076 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:31.092050    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx"
Nov 01 05:47:31.193272 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:31.193185    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:47:31.902132 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:31.902106    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx"
Nov 01 05:47:31.902978 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:31.902956    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx"
Nov 01 05:47:32.323313 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:32.323233    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-57896c8f89-hx4zg"]
Nov 01 05:47:32.328196 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:32.328171    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-57896c8f89-hx4zg"
Nov 01 05:47:32.382942 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:32.382892    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 01 05:47:32.383251 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:32.383230    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-d7505c667e98\""
Nov 01 05:47:32.383384 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:32.383290    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 01 05:47:32.386093 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:32.386073    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-57896c8f89-hx4zg"]
Nov 01 05:47:32.410392 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:32.410367    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/db6aba75-04c5-4352-8c7f-8966a223af4b-provider-credentials\") pod \"payload-processing-external-model-57896c8f89-hx4zg\" (UID: \"db6aba75-04c5-4352-8c7f-8966a223af4b\") " pod="models-as-a-service/payload-processing-external-model-57896c8f89-hx4zg"
Nov 01 05:47:32.410567 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:32.410432    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgz5v\" (UniqueName: \"kubernetes.io/projected/db6aba75-04c5-4352-8c7f-8966a223af4b-kube-api-access-kgz5v\") pod \"payload-processing-external-model-57896c8f89-hx4zg\" (UID: \"db6aba75-04c5-4352-8c7f-8966a223af4b\") " pod="models-as-a-service/payload-processing-external-model-57896c8f89-hx4zg"
Nov 01 05:47:32.410666 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:32.410645    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db6aba75-04c5-4352-8c7f-8966a223af4b-plugins-config-volume\") pod \"payload-processing-external-model-57896c8f89-hx4zg\" (UID: \"db6aba75-04c5-4352-8c7f-8966a223af4b\") " pod="models-as-a-service/payload-processing-external-model-57896c8f89-hx4zg"
Nov 01 05:47:32.410786 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:32.410772    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/db6aba75-04c5-4352-8c7f-8966a223af4b-routing-overlay-volume\") pod \"payload-processing-external-model-57896c8f89-hx4zg\" (UID: \"db6aba75-04c5-4352-8c7f-8966a223af4b\") " pod="models-as-a-service/payload-processing-external-model-57896c8f89-hx4zg"
Nov 01 05:47:32.432324 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:32.432297    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 01 05:47:32.432324 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:32.432304    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-0c80b87999df\""
Nov 01 05:47:32.432542 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:32.432356    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-s59hh\""
Nov 01 05:47:32.432542 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:32.432391    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 01 05:47:32.511844 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:32.511805    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/db6aba75-04c5-4352-8c7f-8966a223af4b-routing-overlay-volume\") pod \"payload-processing-external-model-57896c8f89-hx4zg\" (UID: \"db6aba75-04c5-4352-8c7f-8966a223af4b\") " pod="models-as-a-service/payload-processing-external-model-57896c8f89-hx4zg"
Nov 01 05:47:32.511992 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:32.511861    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/db6aba75-04c5-4352-8c7f-8966a223af4b-provider-credentials\") pod \"payload-processing-external-model-57896c8f89-hx4zg\" (UID: \"db6aba75-04c5-4352-8c7f-8966a223af4b\") " pod="models-as-a-service/payload-processing-external-model-57896c8f89-hx4zg"
Nov 01 05:47:32.511992 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:32.511912    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgz5v\" (UniqueName: \"kubernetes.io/projected/db6aba75-04c5-4352-8c7f-8966a223af4b-kube-api-access-kgz5v\") pod \"payload-processing-external-model-57896c8f89-hx4zg\" (UID: \"db6aba75-04c5-4352-8c7f-8966a223af4b\") " pod="models-as-a-service/payload-processing-external-model-57896c8f89-hx4zg"
Nov 01 05:47:32.512114 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:32.511952    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db6aba75-04c5-4352-8c7f-8966a223af4b-plugins-config-volume\") pod \"payload-processing-external-model-57896c8f89-hx4zg\" (UID: \"db6aba75-04c5-4352-8c7f-8966a223af4b\") " pod="models-as-a-service/payload-processing-external-model-57896c8f89-hx4zg"
Nov 01 05:47:32.512541 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:32.512518    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/db6aba75-04c5-4352-8c7f-8966a223af4b-routing-overlay-volume\") pod \"payload-processing-external-model-57896c8f89-hx4zg\" (UID: \"db6aba75-04c5-4352-8c7f-8966a223af4b\") " pod="models-as-a-service/payload-processing-external-model-57896c8f89-hx4zg"
Nov 01 05:47:32.512679 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:32.512658    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db6aba75-04c5-4352-8c7f-8966a223af4b-plugins-config-volume\") pod \"payload-processing-external-model-57896c8f89-hx4zg\" (UID: \"db6aba75-04c5-4352-8c7f-8966a223af4b\") " pod="models-as-a-service/payload-processing-external-model-57896c8f89-hx4zg"
Nov 01 05:47:32.514487 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:32.514435    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/db6aba75-04c5-4352-8c7f-8966a223af4b-provider-credentials\") pod \"payload-processing-external-model-57896c8f89-hx4zg\" (UID: \"db6aba75-04c5-4352-8c7f-8966a223af4b\") " pod="models-as-a-service/payload-processing-external-model-57896c8f89-hx4zg"
Nov 01 05:47:32.572108 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:32.572082    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgz5v\" (UniqueName: \"kubernetes.io/projected/db6aba75-04c5-4352-8c7f-8966a223af4b-kube-api-access-kgz5v\") pod \"payload-processing-external-model-57896c8f89-hx4zg\" (UID: \"db6aba75-04c5-4352-8c7f-8966a223af4b\") " pod="models-as-a-service/payload-processing-external-model-57896c8f89-hx4zg"
Nov 01 05:47:32.648292 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:32.648274    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-57896c8f89-hx4zg"
Nov 01 05:47:33.022694 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:33.022669    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-57896c8f89-hx4zg"]
Nov 01 05:47:33.025503 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:47:33.025442    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb6aba75_04c5_4352_8c7f_8966a223af4b.slice/crio-16409db6cd66a4f998f65716896afaba9efde6fb494db273856defc44b65d16b WatchSource:0}: Error finding container 16409db6cd66a4f998f65716896afaba9efde6fb494db273856defc44b65d16b: Status 404 returned error can't find the container with id 16409db6cd66a4f998f65716896afaba9efde6fb494db273856defc44b65d16b
Nov 01 05:47:33.837950 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:33.837906    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:47:33.914910 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:33.914876    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-57896c8f89-hx4zg" event={"ID":"db6aba75-04c5-4352-8c7f-8966a223af4b","Type":"ContainerStarted","Data":"f05602983d23ffed79fb868010a4534e4b1733b17158fbb16b8c2d94b1ba6b5b"}
Nov 01 05:47:33.915048 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:33.914937    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-57896c8f89-hx4zg" event={"ID":"db6aba75-04c5-4352-8c7f-8966a223af4b","Type":"ContainerStarted","Data":"16409db6cd66a4f998f65716896afaba9efde6fb494db273856defc44b65d16b"}
Nov 01 05:47:33.915197 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:33.915177    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-57896c8f89-hx4zg"
Nov 01 05:47:34.047314 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:34.047282    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:47:34.075298 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:34.075251    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-57896c8f89-hx4zg" podStartSLOduration=2.075237931 podStartE2EDuration="2.075237931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:47:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:47:34.074331961 +0000 UTC m=+2632.145128866" watchObservedRunningTime="2026-11-01 05:47:34.075237931 +0000 UTC m=+2632.146034650"
Nov 01 05:47:34.293212 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:34.293171    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:47:34.848601 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:34.848572    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:47:37.044353 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:37.044322    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:47:37.227424 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:37.227378    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:47:40.148741 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:40.148705    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:47:42.374799 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:42.374770    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:47:44.970670 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:44.970636    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-57896c8f89-hx4zg"
Nov 01 05:47:45.691478 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:45.690839    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:47:50.235196 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:50.235165    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:47:50.419477 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:50.411917    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:47:50.833624 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:50.833595    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:47:51.051026 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:51.050991    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:47:51.335806 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:51.335727    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:47:56.116053 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:56.116015    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:47:56.627118 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:56.627067    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:47:56.809831 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:47:56.809795    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:48:00.071600 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:48:00.071565    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:48:01.301691 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:48:01.301656    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:48:03.463417 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:48:03.463328    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:48:03.657537 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:48:03.657399    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:48:04.278587 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:48:04.278554    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-848b684699-zqxjt"]
Nov 01 05:48:04.283683 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:48:04.283662    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-848b684699-zqxjt"
Nov 01 05:48:04.308620 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:48:04.308597    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b71980e-1fe5-4331-afc7-0a5e3f4e7148-plugins-config-volume\") pod \"payload-processing-external-model-848b684699-zqxjt\" (UID: \"9b71980e-1fe5-4331-afc7-0a5e3f4e7148\") " pod="models-as-a-service/payload-processing-external-model-848b684699-zqxjt"
Nov 01 05:48:04.308735 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:48:04.308677    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xvrl\" (UniqueName: \"kubernetes.io/projected/9b71980e-1fe5-4331-afc7-0a5e3f4e7148-kube-api-access-8xvrl\") pod \"payload-processing-external-model-848b684699-zqxjt\" (UID: \"9b71980e-1fe5-4331-afc7-0a5e3f4e7148\") " pod="models-as-a-service/payload-processing-external-model-848b684699-zqxjt"
Nov 01 05:48:04.308784 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:48:04.308753    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9b71980e-1fe5-4331-afc7-0a5e3f4e7148-routing-overlay-volume\") pod \"payload-processing-external-model-848b684699-zqxjt\" (UID: \"9b71980e-1fe5-4331-afc7-0a5e3f4e7148\") " pod="models-as-a-service/payload-processing-external-model-848b684699-zqxjt"
Nov 01 05:48:04.308820 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:48:04.308781    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9b71980e-1fe5-4331-afc7-0a5e3f4e7148-provider-credentials\") pod \"payload-processing-external-model-848b684699-zqxjt\" (UID: \"9b71980e-1fe5-4331-afc7-0a5e3f4e7148\") " pod="models-as-a-service/payload-processing-external-model-848b684699-zqxjt"
Nov 01 05:48:04.341247 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:48:04.341215    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-848b684699-zqxjt"]
Nov 01 05:48:04.438510 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:48:04.415085    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xvrl\" (UniqueName: \"kubernetes.io/projected/9b71980e-1fe5-4331-afc7-0a5e3f4e7148-kube-api-access-8xvrl\") pod \"payload-processing-external-model-848b684699-zqxjt\" (UID: \"9b71980e-1fe5-4331-afc7-0a5e3f4e7148\") " pod="models-as-a-service/payload-processing-external-model-848b684699-zqxjt"
Nov 01 05:48:04.438510 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:48:04.415192    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9b71980e-1fe5-4331-afc7-0a5e3f4e7148-routing-overlay-volume\") pod \"payload-processing-external-model-848b684699-zqxjt\" (UID: \"9b71980e-1fe5-4331-afc7-0a5e3f4e7148\") " pod="models-as-a-service/payload-processing-external-model-848b684699-zqxjt"
Nov 01 05:48:04.438510 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:48:04.415250    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9b71980e-1fe5-4331-afc7-0a5e3f4e7148-provider-credentials\") pod \"payload-processing-external-model-848b684699-zqxjt\" (UID: \"9b71980e-1fe5-4331-afc7-0a5e3f4e7148\") " pod="models-as-a-service/payload-processing-external-model-848b684699-zqxjt"
Nov 01 05:48:04.438510 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:48:04.415444    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b71980e-1fe5-4331-afc7-0a5e3f4e7148-plugins-config-volume\") pod \"payload-processing-external-model-848b684699-zqxjt\" (UID: \"9b71980e-1fe5-4331-afc7-0a5e3f4e7148\") " pod="models-as-a-service/payload-processing-external-model-848b684699-zqxjt"
Nov 01 05:48:04.438510 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:48:04.416184    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9b71980e-1fe5-4331-afc7-0a5e3f4e7148-routing-overlay-volume\") pod \"payload-processing-external-model-848b684699-zqxjt\" (UID: \"9b71980e-1fe5-4331-afc7-0a5e3f4e7148\") " pod="models-as-a-service/payload-processing-external-model-848b684699-zqxjt"
Nov 01 05:48:04.438510 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:48:04.434288    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b71980e-1fe5-4331-afc7-0a5e3f4e7148-plugins-config-volume\") pod \"payload-processing-external-model-848b684699-zqxjt\" (UID: \"9b71980e-1fe5-4331-afc7-0a5e3f4e7148\") " pod="models-as-a-service/payload-processing-external-model-848b684699-zqxjt"
Nov 01 05:48:04.446478 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:48:04.445358    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9b71980e-1fe5-4331-afc7-0a5e3f4e7148-provider-credentials\") pod \"payload-processing-external-model-848b684699-zqxjt\" (UID: \"9b71980e-1fe5-4331-afc7-0a5e3f4e7148\") " pod="models-as-a-service/payload-processing-external-model-848b684699-zqxjt"
Nov 01 05:48:04.480101 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:48:04.480069    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xvrl\" (UniqueName: \"kubernetes.io/projected/9b71980e-1fe5-4331-afc7-0a5e3f4e7148-kube-api-access-8xvrl\") pod \"payload-processing-external-model-848b684699-zqxjt\" (UID: \"9b71980e-1fe5-4331-afc7-0a5e3f4e7148\") " pod="models-as-a-service/payload-processing-external-model-848b684699-zqxjt"
Nov 01 05:48:04.597730 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:48:04.597658    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-848b684699-zqxjt"
Nov 01 05:48:04.973628 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:48:04.973601    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-848b684699-zqxjt"]
Nov 01 05:48:04.975684 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:48:04.975657    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b71980e_1fe5_4331_afc7_0a5e3f4e7148.slice/crio-be05d01e88c1cfa8429c206744d14cbbb2605addff1f2de67213e1850b54efb7 WatchSource:0}: Error finding container be05d01e88c1cfa8429c206744d14cbbb2605addff1f2de67213e1850b54efb7: Status 404 returned error can't find the container with id be05d01e88c1cfa8429c206744d14cbbb2605addff1f2de67213e1850b54efb7
Nov 01 05:48:05.191294 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:48:05.191260    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-848b684699-zqxjt" event={"ID":"9b71980e-1fe5-4331-afc7-0a5e3f4e7148","Type":"ContainerStarted","Data":"e0a7b927800c3d9d0042f46503282bb007e8358da522ce669a4e477d4f68c480"}
Nov 01 05:48:05.191476 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:48:05.191316    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-848b684699-zqxjt"
Nov 01 05:48:05.191476 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:48:05.191334    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-848b684699-zqxjt" event={"ID":"9b71980e-1fe5-4331-afc7-0a5e3f4e7148","Type":"ContainerStarted","Data":"be05d01e88c1cfa8429c206744d14cbbb2605addff1f2de67213e1850b54efb7"}
Nov 01 05:48:05.308934 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:48:05.308842    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-848b684699-zqxjt" podStartSLOduration=1.30882814 podStartE2EDuration="1.30882814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:48:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:48:05.306319168 +0000 UTC m=+2663.377115885" watchObservedRunningTime="2026-11-01 05:48:05.30882814 +0000 UTC m=+2663.379624885"
Nov 01 05:48:06.107832 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:48:06.107804    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:48:08.786651 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:48:08.786604    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:48:10.907582 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:48:10.907548    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:48:12.119492 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:48:12.119423    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:48:16.004052 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:48:16.004021    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:48:16.199670 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:48:16.199625    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-848b684699-zqxjt" podUID="9b71980e-1fe5-4331-afc7-0a5e3f4e7148" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 05:48:20.764847 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:48:20.764805    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:48:25.530139 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:48:25.530103    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:48:26.198729 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:48:26.198693    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-848b684699-zqxjt" podUID="9b71980e-1fe5-4331-afc7-0a5e3f4e7148" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 05:48:30.069999 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:48:30.069965    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:48:30.249568 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:48:30.249533    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:48:34.952276 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:48:34.952240    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:48:35.143768 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:48:35.143731    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:48:35.326180 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:48:35.326113    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:48:36.197747 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:48:36.197713    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-848b684699-zqxjt" podUID="9b71980e-1fe5-4331-afc7-0a5e3f4e7148" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 05:48:39.925053 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:48:39.924999    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:48:44.824114 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:48:44.824068    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:48:45.439232 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:48:45.439199    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-848b684699-zqxjt"]
Nov 01 05:48:45.440530 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:48:45.440503    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-848b684699-zqxjt" podUID="9b71980e-1fe5-4331-afc7-0a5e3f4e7148" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 05:48:49.019143 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:48:49.019100    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:48:49.618903 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:48:49.618837    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:48:50.441796 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:48:50.441544    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-848b684699-zqxjt" podUID="9b71980e-1fe5-4331-afc7-0a5e3f4e7148" containerName="payload-processing" containerID="cri-o://e0a7b927800c3d9d0042f46503282bb007e8358da522ce669a4e477d4f68c480" gracePeriod=25
Nov 01 05:48:50.968386 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:48:50.968328    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-848b684699-zqxjt"
Nov 01 05:48:51.105533 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:48:51.104940    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b71980e-1fe5-4331-afc7-0a5e3f4e7148-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b71980e-1fe5-4331-afc7-0a5e3f4e7148-plugins-config-volume\") pod \"9b71980e-1fe5-4331-afc7-0a5e3f4e7148\" (UID: \"9b71980e-1fe5-4331-afc7-0a5e3f4e7148\") "
Nov 01 05:48:51.105533 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:48:51.105079    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b71980e-1fe5-4331-afc7-0a5e3f4e7148-kube-api-access-8xvrl\" (UniqueName: \"kubernetes.io/projected/9b71980e-1fe5-4331-afc7-0a5e3f4e7148-kube-api-access-8xvrl\") pod \"9b71980e-1fe5-4331-afc7-0a5e3f4e7148\" (UID: \"9b71980e-1fe5-4331-afc7-0a5e3f4e7148\") "
Nov 01 05:48:51.105533 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:48:51.105132    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b71980e-1fe5-4331-afc7-0a5e3f4e7148-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9b71980e-1fe5-4331-afc7-0a5e3f4e7148-provider-credentials\") pod \"9b71980e-1fe5-4331-afc7-0a5e3f4e7148\" (UID: \"9b71980e-1fe5-4331-afc7-0a5e3f4e7148\") "
Nov 01 05:48:51.105533 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:48:51.105185    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b71980e-1fe5-4331-afc7-0a5e3f4e7148-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9b71980e-1fe5-4331-afc7-0a5e3f4e7148-routing-overlay-volume\") pod \"9b71980e-1fe5-4331-afc7-0a5e3f4e7148\" (UID: \"9b71980e-1fe5-4331-afc7-0a5e3f4e7148\") "
Nov 01 05:48:51.106483 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:48:51.106397    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b71980e-1fe5-4331-afc7-0a5e3f4e7148-plugins-config-volume" pod "9b71980e-1fe5-4331-afc7-0a5e3f4e7148" (UID: "9b71980e-1fe5-4331-afc7-0a5e3f4e7148"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:48:51.126573 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:48:51.107888    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b71980e-1fe5-4331-afc7-0a5e3f4e7148-routing-overlay-volume" pod "9b71980e-1fe5-4331-afc7-0a5e3f4e7148" (UID: "9b71980e-1fe5-4331-afc7-0a5e3f4e7148"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:48:51.126573 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:48:51.116920    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b71980e-1fe5-4331-afc7-0a5e3f4e7148-kube-api-access-8xvrl" pod "9b71980e-1fe5-4331-afc7-0a5e3f4e7148" (UID: "9b71980e-1fe5-4331-afc7-0a5e3f4e7148"). InnerVolumeSpecName "kube-api-access-8xvrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:48:51.165385 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:48:51.165325    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b71980e-1fe5-4331-afc7-0a5e3f4e7148-provider-credentials" pod "9b71980e-1fe5-4331-afc7-0a5e3f4e7148" (UID: "9b71980e-1fe5-4331-afc7-0a5e3f4e7148"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:48:51.209570 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:48:51.209445    5689 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b71980e-1fe5-4331-afc7-0a5e3f4e7148-plugins-config-volume\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:48:51.209570 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:48:51.209506    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xvrl\" (UniqueName: \"kubernetes.io/projected/9b71980e-1fe5-4331-afc7-0a5e3f4e7148-kube-api-access-8xvrl\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:48:51.209570 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:48:51.209522    5689 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9b71980e-1fe5-4331-afc7-0a5e3f4e7148-provider-credentials\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:48:51.209570 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:48:51.209540    5689 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9b71980e-1fe5-4331-afc7-0a5e3f4e7148-routing-overlay-volume\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:48:51.573484 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:48:51.573003    5689 generic.go:356] "Generic (PLEG): container finished" podID="9b71980e-1fe5-4331-afc7-0a5e3f4e7148" containerID="e0a7b927800c3d9d0042f46503282bb007e8358da522ce669a4e477d4f68c480" exitCode=0
Nov 01 05:48:51.573484 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:48:51.573202    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-848b684699-zqxjt" event={"ID":"9b71980e-1fe5-4331-afc7-0a5e3f4e7148","Type":"ContainerDied","Data":"e0a7b927800c3d9d0042f46503282bb007e8358da522ce669a4e477d4f68c480"}
Nov 01 05:48:51.573484 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:48:51.573257    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-848b684699-zqxjt" event={"ID":"9b71980e-1fe5-4331-afc7-0a5e3f4e7148","Type":"ContainerDied","Data":"be05d01e88c1cfa8429c206744d14cbbb2605addff1f2de67213e1850b54efb7"}
Nov 01 05:48:51.573484 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:48:51.573290    5689 scope.go:122] "RemoveContainer" containerID="e0a7b927800c3d9d0042f46503282bb007e8358da522ce669a4e477d4f68c480"
Nov 01 05:48:51.574170 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:48:51.573493    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-848b684699-zqxjt"
Nov 01 05:48:51.710982 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:48:51.710629    5689 scope.go:122] "RemoveContainer" containerID="e0a7b927800c3d9d0042f46503282bb007e8358da522ce669a4e477d4f68c480"
Nov 01 05:48:51.711333 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:48:51.711237    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0a7b927800c3d9d0042f46503282bb007e8358da522ce669a4e477d4f68c480\": container with ID starting with e0a7b927800c3d9d0042f46503282bb007e8358da522ce669a4e477d4f68c480 not found: ID does not exist" containerID="e0a7b927800c3d9d0042f46503282bb007e8358da522ce669a4e477d4f68c480"
Nov 01 05:48:51.711333 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:48:51.711278    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0a7b927800c3d9d0042f46503282bb007e8358da522ce669a4e477d4f68c480"} err="failed to get container status \"e0a7b927800c3d9d0042f46503282bb007e8358da522ce669a4e477d4f68c480\": rpc error: code = NotFound desc = could not find container \"e0a7b927800c3d9d0042f46503282bb007e8358da522ce669a4e477d4f68c480\": container with ID starting with e0a7b927800c3d9d0042f46503282bb007e8358da522ce669a4e477d4f68c480 not found: ID does not exist"
Nov 01 05:48:51.794480 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:48:51.793524    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-848b684699-zqxjt"]
Nov 01 05:48:51.851535 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:48:51.850997    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-848b684699-zqxjt"]
Nov 01 05:48:52.112026 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:48:52.111944    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b71980e-1fe5-4331-afc7-0a5e3f4e7148" path="/var/lib/kubelet/pods/9b71980e-1fe5-4331-afc7-0a5e3f4e7148/volumes"
Nov 01 05:48:52.628593 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:48:52.628533    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a48099/simulator-w4-a48099-kserve-7df86f7d-rdlrm" podUID="12c3ad65-a804-429f-ab42-9cc7b2433855" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 05:48:52.673258 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:48:52.673218    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3d70ff/distinct-2-w2-3d70ff-kserve-868b775cd4-nhx4f" podUID="0c79347d-eb2e-49b2-af27-48ddcf86b7bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 05:48:52.960479 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:48:52.959531    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7d1ec3/distinct-w0-7d1ec3-kserve-58cdfc8d58-n7xpt" podUID="4a1fcaaf-0b04-4602-8985-866f6152303d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded"
Nov 01 05:48:54.607588 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:48:54.607556    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:48:56.693515 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:48:56.693440    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:48:56.867611 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:48:56.867580    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:48:59.697768 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:48:59.697729    5689 generic.go:356] "Generic (PLEG): container finished" podID="9d69cb72-f4e4-4ef1-9ceb-210cbd671a54" containerID="db8bd7e6c8fc393881c66a58e1d3a50e711cf8f9542a92a6752657aac397097b" exitCode=0
Nov 01 05:48:59.698186 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:48:59.697794    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx" event={"ID":"9d69cb72-f4e4-4ef1-9ceb-210cbd671a54","Type":"ContainerDied","Data":"db8bd7e6c8fc393881c66a58e1d3a50e711cf8f9542a92a6752657aac397097b"}
Nov 01 05:48:59.698244 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:48:59.698227    5689 scope.go:122] "RemoveContainer" containerID="db8bd7e6c8fc393881c66a58e1d3a50e711cf8f9542a92a6752657aac397097b"
Nov 01 05:48:59.699695 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:48:59.699670    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:48:59.699776 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:48:59.699741    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:48:59.699813 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:48:59.699790    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:48:59.701708 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:48:59.701686    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:48:59.883129 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:48:59.883046    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:49:00.071346 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:49:00.071310    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:49:00.708158 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:00.708126    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx" event={"ID":"9d69cb72-f4e4-4ef1-9ceb-210cbd671a54","Type":"ContainerStarted","Data":"7c9ff3969a214671a9128620947f7f6691626fcd9fc594c29d3568c4a79e9642"}
Nov 01 05:49:01.099537 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:01.097425    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx"
Nov 01 05:49:01.099537 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:01.097645    5689 patch_prober.go:29] interesting pod/e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Nov 01 05:49:01.099537 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:01.097690    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx" podUID="9d69cb72-f4e4-4ef1-9ceb-210cbd671a54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Nov 01 05:49:01.902715 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:01.902683    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx"
Nov 01 05:49:02.092477 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:02.089874    5689 patch_prober.go:29] interesting pod/e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Nov 01 05:49:02.092477 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:02.089928    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx" podUID="9d69cb72-f4e4-4ef1-9ceb-210cbd671a54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Nov 01 05:49:02.178523 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:49:02.176217    5689 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-0c80b87999df: secret "agc-cred-0c80b87999df" not found
Nov 01 05:49:02.178523 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:49:02.176249    5689 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-57896c8f89-hx4zg: secret "agc-cred-0c80b87999df" not found
Nov 01 05:49:02.178523 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:49:02.176301    5689 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db6aba75-04c5-4352-8c7f-8966a223af4b-provider-credentials podName:db6aba75-04c5-4352-8c7f-8966a223af4b nodeName:}" failed. No retries permitted until 2026-11-01 05:49:02.676283126 +0000 UTC m=+2720.747079842 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/db6aba75-04c5-4352-8c7f-8966a223af4b-provider-credentials") pod "payload-processing-external-model-57896c8f89-hx4zg" (UID: "db6aba75-04c5-4352-8c7f-8966a223af4b") : secret "agc-cred-0c80b87999df" not found
Nov 01 05:49:02.703811 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:49:02.703361    5689 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-0c80b87999df: secret "agc-cred-0c80b87999df" not found
Nov 01 05:49:02.703811 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:49:02.703394    5689 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-57896c8f89-hx4zg: secret "agc-cred-0c80b87999df" not found
Nov 01 05:49:02.703811 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:49:02.703468    5689 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db6aba75-04c5-4352-8c7f-8966a223af4b-provider-credentials podName:db6aba75-04c5-4352-8c7f-8966a223af4b nodeName:}" failed. No retries permitted until 2026-11-01 05:49:03.70343308 +0000 UTC m=+2721.774229793 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/db6aba75-04c5-4352-8c7f-8966a223af4b-provider-credentials") pod "payload-processing-external-model-57896c8f89-hx4zg" (UID: "db6aba75-04c5-4352-8c7f-8966a223af4b") : secret "agc-cred-0c80b87999df" not found
Nov 01 05:49:02.944090 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:02.944059    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:49:03.088892 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:03.088798    5689 patch_prober.go:29] interesting pod/e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Nov 01 05:49:03.089157 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:03.089128    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx" podUID="9d69cb72-f4e4-4ef1-9ceb-210cbd671a54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Nov 01 05:49:03.717129 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:49:03.717094    5689 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-0c80b87999df: secret "agc-cred-0c80b87999df" not found
Nov 01 05:49:03.717129 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:49:03.717122    5689 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-57896c8f89-hx4zg: secret "agc-cred-0c80b87999df" not found
Nov 01 05:49:03.717345 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:49:03.717177    5689 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db6aba75-04c5-4352-8c7f-8966a223af4b-provider-credentials podName:db6aba75-04c5-4352-8c7f-8966a223af4b nodeName:}" failed. No retries permitted until 2026-11-01 05:49:05.717160701 +0000 UTC m=+2723.787957423 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/db6aba75-04c5-4352-8c7f-8966a223af4b-provider-credentials") pod "payload-processing-external-model-57896c8f89-hx4zg" (UID: "db6aba75-04c5-4352-8c7f-8966a223af4b") : secret "agc-cred-0c80b87999df" not found
Nov 01 05:49:04.099530 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:04.097822    5689 patch_prober.go:29] interesting pod/e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Nov 01 05:49:04.099530 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:04.097887    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx" podUID="9d69cb72-f4e4-4ef1-9ceb-210cbd671a54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Nov 01 05:49:04.623530 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:04.623495    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:49:05.096476 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:05.095780    5689 patch_prober.go:29] interesting pod/e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Nov 01 05:49:05.096476 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:05.095838    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx" podUID="9d69cb72-f4e4-4ef1-9ceb-210cbd671a54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Nov 01 05:49:05.755288 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:49:05.755255    5689 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-0c80b87999df: secret "agc-cred-0c80b87999df" not found
Nov 01 05:49:05.755288 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:49:05.755282    5689 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-57896c8f89-hx4zg: secret "agc-cred-0c80b87999df" not found
Nov 01 05:49:05.755930 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:49:05.755341    5689 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db6aba75-04c5-4352-8c7f-8966a223af4b-provider-credentials podName:db6aba75-04c5-4352-8c7f-8966a223af4b nodeName:}" failed. No retries permitted until 2026-11-01 05:49:09.755321347 +0000 UTC m=+2727.826118061 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/db6aba75-04c5-4352-8c7f-8966a223af4b-provider-credentials") pod "payload-processing-external-model-57896c8f89-hx4zg" (UID: "db6aba75-04c5-4352-8c7f-8966a223af4b") : secret "agc-cred-0c80b87999df" not found
Nov 01 05:49:06.089698 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:06.089619    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx"
Nov 01 05:49:06.090531 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:06.090511    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx"
Nov 01 05:49:06.623234 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:06.623202    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:49:06.802752 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:06.802721    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:49:06.991014 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:06.990984    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:49:09.140358 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:09.140322    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:49:09.439509 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:09.439444    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:49:09.814405 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:49:09.814331    5689 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-0c80b87999df: secret "agc-cred-0c80b87999df" not found
Nov 01 05:49:09.814405 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:49:09.814357    5689 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-57896c8f89-hx4zg: secret "agc-cred-0c80b87999df" not found
Nov 01 05:49:09.814596 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:49:09.814423    5689 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db6aba75-04c5-4352-8c7f-8966a223af4b-provider-credentials podName:db6aba75-04c5-4352-8c7f-8966a223af4b nodeName:}" failed. No retries permitted until 2026-11-01 05:49:17.814406755 +0000 UTC m=+2735.885203456 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/db6aba75-04c5-4352-8c7f-8966a223af4b-provider-credentials") pod "payload-processing-external-model-57896c8f89-hx4zg" (UID: "db6aba75-04c5-4352-8c7f-8966a223af4b") : secret "agc-cred-0c80b87999df" not found
Nov 01 05:49:14.322013 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:14.321978    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:49:15.540438 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:15.540406    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:49:17.842478 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:49:17.841512    5689 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-0c80b87999df: secret "agc-cred-0c80b87999df" not found
Nov 01 05:49:17.842478 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:49:17.841552    5689 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-57896c8f89-hx4zg: secret "agc-cred-0c80b87999df" not found
Nov 01 05:49:17.842478 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:49:17.841625    5689 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db6aba75-04c5-4352-8c7f-8966a223af4b-provider-credentials podName:db6aba75-04c5-4352-8c7f-8966a223af4b nodeName:}" failed. No retries permitted until 2026-11-01 05:49:33.841605691 +0000 UTC m=+2751.912402407 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/db6aba75-04c5-4352-8c7f-8966a223af4b-provider-credentials") pod "payload-processing-external-model-57896c8f89-hx4zg" (UID: "db6aba75-04c5-4352-8c7f-8966a223af4b") : secret "agc-cred-0c80b87999df" not found
Nov 01 05:49:18.632479 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:18.629552    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-zml98 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 05:49:18.632479 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:18.629612    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zml98" podUID="758d79fd-92ca-46fb-b487-694dc9818b61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 05:49:19.388501 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:19.388447    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:49:24.205370 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:24.205318    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:49:25.630802 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:25.630767    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-57896c8f89-hx4zg"]
Nov 01 05:49:29.036576 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:29.036537    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:49:30.069149 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:49:30.069116    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:49:30.631433 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:30.631398    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-57896c8f89-hx4zg" podUID="db6aba75-04c5-4352-8c7f-8966a223af4b" containerName="payload-processing" containerID="cri-o://f05602983d23ffed79fb868010a4534e4b1733b17158fbb16b8c2d94b1ba6b5b" gracePeriod=25
Nov 01 05:49:30.906410 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:30.906377    5689 generic.go:356] "Generic (PLEG): container finished" podID="db6aba75-04c5-4352-8c7f-8966a223af4b" containerID="f05602983d23ffed79fb868010a4534e4b1733b17158fbb16b8c2d94b1ba6b5b" exitCode=0
Nov 01 05:49:30.906553 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:30.906484    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-57896c8f89-hx4zg" event={"ID":"db6aba75-04c5-4352-8c7f-8966a223af4b","Type":"ContainerDied","Data":"f05602983d23ffed79fb868010a4534e4b1733b17158fbb16b8c2d94b1ba6b5b"}
Nov 01 05:49:30.906600 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:30.906548    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-57896c8f89-hx4zg" event={"ID":"db6aba75-04c5-4352-8c7f-8966a223af4b","Type":"ContainerDied","Data":"16409db6cd66a4f998f65716896afaba9efde6fb494db273856defc44b65d16b"}
Nov 01 05:49:30.906600 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:30.906574    5689 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16409db6cd66a4f998f65716896afaba9efde6fb494db273856defc44b65d16b"
Nov 01 05:49:30.910358 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:30.910341    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-57896c8f89-hx4zg"
Nov 01 05:49:30.955727 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:30.955702    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db6aba75-04c5-4352-8c7f-8966a223af4b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/db6aba75-04c5-4352-8c7f-8966a223af4b-provider-credentials\") pod \"db6aba75-04c5-4352-8c7f-8966a223af4b\" (UID: \"db6aba75-04c5-4352-8c7f-8966a223af4b\") "
Nov 01 05:49:30.955844 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:30.955777    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db6aba75-04c5-4352-8c7f-8966a223af4b-kube-api-access-kgz5v\" (UniqueName: \"kubernetes.io/projected/db6aba75-04c5-4352-8c7f-8966a223af4b-kube-api-access-kgz5v\") pod \"db6aba75-04c5-4352-8c7f-8966a223af4b\" (UID: \"db6aba75-04c5-4352-8c7f-8966a223af4b\") "
Nov 01 05:49:30.955844 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:30.955826    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db6aba75-04c5-4352-8c7f-8966a223af4b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/db6aba75-04c5-4352-8c7f-8966a223af4b-routing-overlay-volume\") pod \"db6aba75-04c5-4352-8c7f-8966a223af4b\" (UID: \"db6aba75-04c5-4352-8c7f-8966a223af4b\") "
Nov 01 05:49:30.955997 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:30.955981    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db6aba75-04c5-4352-8c7f-8966a223af4b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db6aba75-04c5-4352-8c7f-8966a223af4b-plugins-config-volume\") pod \"db6aba75-04c5-4352-8c7f-8966a223af4b\" (UID: \"db6aba75-04c5-4352-8c7f-8966a223af4b\") "
Nov 01 05:49:30.956252 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:30.956223    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db6aba75-04c5-4352-8c7f-8966a223af4b-routing-overlay-volume" pod "db6aba75-04c5-4352-8c7f-8966a223af4b" (UID: "db6aba75-04c5-4352-8c7f-8966a223af4b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:49:30.956372 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:30.956325    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db6aba75-04c5-4352-8c7f-8966a223af4b-plugins-config-volume" pod "db6aba75-04c5-4352-8c7f-8966a223af4b" (UID: "db6aba75-04c5-4352-8c7f-8966a223af4b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:49:30.956684 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:30.956665    5689 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db6aba75-04c5-4352-8c7f-8966a223af4b-plugins-config-volume\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:49:30.956785 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:30.956690    5689 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/db6aba75-04c5-4352-8c7f-8966a223af4b-routing-overlay-volume\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:49:30.958081 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:30.958061    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db6aba75-04c5-4352-8c7f-8966a223af4b-kube-api-access-kgz5v" pod "db6aba75-04c5-4352-8c7f-8966a223af4b" (UID: "db6aba75-04c5-4352-8c7f-8966a223af4b"). InnerVolumeSpecName "kube-api-access-kgz5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:49:30.958155 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:30.958133    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db6aba75-04c5-4352-8c7f-8966a223af4b-provider-credentials" pod "db6aba75-04c5-4352-8c7f-8966a223af4b" (UID: "db6aba75-04c5-4352-8c7f-8966a223af4b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:49:31.058131 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:31.058091    5689 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/db6aba75-04c5-4352-8c7f-8966a223af4b-provider-credentials\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:49:31.058131 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:31.058130    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgz5v\" (UniqueName: \"kubernetes.io/projected/db6aba75-04c5-4352-8c7f-8966a223af4b-kube-api-access-kgz5v\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:49:31.912051 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:31.912021    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-57896c8f89-hx4zg"
Nov 01 05:49:32.074701 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:32.074672    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-57896c8f89-hx4zg"]
Nov 01 05:49:32.124047 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:32.124019    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-57896c8f89-hx4zg"]
Nov 01 05:49:33.742673 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:33.742638    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:49:34.068269 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:34.068198    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db6aba75-04c5-4352-8c7f-8966a223af4b" path="/var/lib/kubelet/pods/db6aba75-04c5-4352-8c7f-8966a223af4b/volumes"
Nov 01 05:49:38.495739 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:38.495700    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:49:41.328092 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:41.328057    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:49:42.561495 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:42.560332    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:49:43.631018 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:43.630982    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:49:43.876535 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:43.876423    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:49:44.619848 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:44.619813    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:49:47.870966 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:47.870933    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:49:48.043362 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:48.043330    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:49:48.228427 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:48.228388    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:49:49.666238 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:49.666204    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:49:49.960771 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:49.960705    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:49:50.134137 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:50.134108    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:49:52.210434 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:52.210376    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:49:52.394142 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:52.394096    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:49:52.605625 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:52.605547    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-7a360a-d4b7c96cf-cswr9"]
Nov 01 05:49:52.606081 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:52.606065    5689 cpu_manager.go:414] "containerMap: removing container" podUID="9b71980e-1fe5-4331-afc7-0a5e3f4e7148" containerName="payload-processing"
Nov 01 05:49:52.606081 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:52.606079    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b71980e-1fe5-4331-afc7-0a5e3f4e7148" containerName="payload-processing"
Nov 01 05:49:52.606224 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:52.606103    5689 cpu_manager.go:414] "containerMap: removing container" podUID="db6aba75-04c5-4352-8c7f-8966a223af4b" containerName="payload-processing"
Nov 01 05:49:52.606224 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:52.606108    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db6aba75-04c5-4352-8c7f-8966a223af4b" containerName="payload-processing"
Nov 01 05:49:52.606288 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:52.606223    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b71980e-1fe5-4331-afc7-0a5e3f4e7148" containerName="payload-processing"
Nov 01 05:49:52.606288 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:52.606234    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="db6aba75-04c5-4352-8c7f-8966a223af4b" containerName="payload-processing"
Nov 01 05:49:52.610822 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:52.610806    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7a360a-d4b7c96cf-cswr9"
Nov 01 05:49:52.618528 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:52.618497    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1177a649-2110-4dae-86a9-dc801392d954-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-7a360a-d4b7c96cf-cswr9\" (UID: \"1177a649-2110-4dae-86a9-dc801392d954\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7a360a-d4b7c96cf-cswr9"
Nov 01 05:49:52.618648 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:52.618625    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcwgm\" (UniqueName: \"kubernetes.io/projected/1177a649-2110-4dae-86a9-dc801392d954-kube-api-access-qcwgm\") pod \"payload-pre-processing-e2e-extmod-b-7a360a-d4b7c96cf-cswr9\" (UID: \"1177a649-2110-4dae-86a9-dc801392d954\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7a360a-d4b7c96cf-cswr9"
Nov 01 05:49:52.665185 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:52.665157    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-7a360a-dockercfg-nwk76\""
Nov 01 05:49:52.715715 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:52.715675    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-7a360a\""
Nov 01 05:49:52.735529 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:52.730053    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1177a649-2110-4dae-86a9-dc801392d954-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-7a360a-d4b7c96cf-cswr9\" (UID: \"1177a649-2110-4dae-86a9-dc801392d954\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7a360a-d4b7c96cf-cswr9"
Nov 01 05:49:52.735529 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:52.730147    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcwgm\" (UniqueName: \"kubernetes.io/projected/1177a649-2110-4dae-86a9-dc801392d954-kube-api-access-qcwgm\") pod \"payload-pre-processing-e2e-extmod-b-7a360a-d4b7c96cf-cswr9\" (UID: \"1177a649-2110-4dae-86a9-dc801392d954\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7a360a-d4b7c96cf-cswr9"
Nov 01 05:49:52.735529 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:52.731911    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1177a649-2110-4dae-86a9-dc801392d954-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-7a360a-d4b7c96cf-cswr9\" (UID: \"1177a649-2110-4dae-86a9-dc801392d954\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7a360a-d4b7c96cf-cswr9"
Nov 01 05:49:52.743573 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:52.743535    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-7a360a-d4b7c96cf-cswr9"]
Nov 01 05:49:52.814400 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:52.814366    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcwgm\" (UniqueName: \"kubernetes.io/projected/1177a649-2110-4dae-86a9-dc801392d954-kube-api-access-qcwgm\") pod \"payload-pre-processing-e2e-extmod-b-7a360a-d4b7c96cf-cswr9\" (UID: \"1177a649-2110-4dae-86a9-dc801392d954\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7a360a-d4b7c96cf-cswr9"
Nov 01 05:49:52.816071 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:52.816048    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-7a360a-6cf998897-7v6d7"]
Nov 01 05:49:52.820566 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:52.820542    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-7a360a-6cf998897-7v6d7"
Nov 01 05:49:52.840004 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:52.839975    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-7a360a-6cf998897-7v6d7"]
Nov 01 05:49:52.924548 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:52.924524    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7a360a-d4b7c96cf-cswr9"
Nov 01 05:49:52.932966 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:52.932940    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt2h8\" (UniqueName: \"kubernetes.io/projected/838940c1-785d-4fb4-9c0a-c967b252ff64-kube-api-access-gt2h8\") pod \"payload-processing-e2e-extmod-b-7a360a-6cf998897-7v6d7\" (UID: \"838940c1-785d-4fb4-9c0a-c967b252ff64\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-7a360a-6cf998897-7v6d7"
Nov 01 05:49:52.933066 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:52.932986    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/838940c1-785d-4fb4-9c0a-c967b252ff64-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-7a360a-6cf998897-7v6d7\" (UID: \"838940c1-785d-4fb4-9c0a-c967b252ff64\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-7a360a-6cf998897-7v6d7"
Nov 01 05:49:53.034120 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:53.034088    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt2h8\" (UniqueName: \"kubernetes.io/projected/838940c1-785d-4fb4-9c0a-c967b252ff64-kube-api-access-gt2h8\") pod \"payload-processing-e2e-extmod-b-7a360a-6cf998897-7v6d7\" (UID: \"838940c1-785d-4fb4-9c0a-c967b252ff64\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-7a360a-6cf998897-7v6d7"
Nov 01 05:49:53.034314 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:53.034140    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/838940c1-785d-4fb4-9c0a-c967b252ff64-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-7a360a-6cf998897-7v6d7\" (UID: \"838940c1-785d-4fb4-9c0a-c967b252ff64\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-7a360a-6cf998897-7v6d7"
Nov 01 05:49:53.035035 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:53.035013    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/838940c1-785d-4fb4-9c0a-c967b252ff64-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-7a360a-6cf998897-7v6d7\" (UID: \"838940c1-785d-4fb4-9c0a-c967b252ff64\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-7a360a-6cf998897-7v6d7"
Nov 01 05:49:53.097897 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:53.097871    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt2h8\" (UniqueName: \"kubernetes.io/projected/838940c1-785d-4fb4-9c0a-c967b252ff64-kube-api-access-gt2h8\") pod \"payload-processing-e2e-extmod-b-7a360a-6cf998897-7v6d7\" (UID: \"838940c1-785d-4fb4-9c0a-c967b252ff64\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-7a360a-6cf998897-7v6d7"
Nov 01 05:49:53.139392 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:53.139364    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-7a360a-6cf998897-7v6d7"
Nov 01 05:49:53.304241 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:53.304216    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-7a360a-d4b7c96cf-cswr9"]
Nov 01 05:49:53.306385 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:49:53.306359    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1177a649_2110_4dae_86a9_dc801392d954.slice/crio-bfa908e32db75eef7e600ce051fdd42a976b60ebc4fec6749173d67383d58896 WatchSource:0}: Error finding container bfa908e32db75eef7e600ce051fdd42a976b60ebc4fec6749173d67383d58896: Status 404 returned error can't find the container with id bfa908e32db75eef7e600ce051fdd42a976b60ebc4fec6749173d67383d58896
Nov 01 05:49:53.516560 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:53.516537    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-7a360a-6cf998897-7v6d7"]
Nov 01 05:49:53.518819 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:49:53.518795    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod838940c1_785d_4fb4_9c0a_c967b252ff64.slice/crio-81849d357325aee59495b078d29e225047f54fd96efe5cbd6f3c1d6798b25ac9 WatchSource:0}: Error finding container 81849d357325aee59495b078d29e225047f54fd96efe5cbd6f3c1d6798b25ac9: Status 404 returned error can't find the container with id 81849d357325aee59495b078d29e225047f54fd96efe5cbd6f3c1d6798b25ac9
Nov 01 05:49:54.025676 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:54.025617    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:49:54.122930 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:54.122889    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-7a360a-6cf998897-7v6d7" event={"ID":"838940c1-785d-4fb4-9c0a-c967b252ff64","Type":"ContainerStarted","Data":"1dcc327fee13764dfbf2d2ea79754c68b9ee33654115411bc0791e444b04846b"}
Nov 01 05:49:54.123123 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:54.122947    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-7a360a-6cf998897-7v6d7" event={"ID":"838940c1-785d-4fb4-9c0a-c967b252ff64","Type":"ContainerStarted","Data":"81849d357325aee59495b078d29e225047f54fd96efe5cbd6f3c1d6798b25ac9"}
Nov 01 05:49:54.123123 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:54.123108    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-7a360a-6cf998897-7v6d7"
Nov 01 05:49:54.124576 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:54.124550    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7a360a-d4b7c96cf-cswr9" event={"ID":"1177a649-2110-4dae-86a9-dc801392d954","Type":"ContainerStarted","Data":"ab76d949771bfee429a81a4f4622e1e3551be494f932402b0f9f0d3725bfd002"}
Nov 01 05:49:54.124723 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:54.124596    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7a360a-d4b7c96cf-cswr9" event={"ID":"1177a649-2110-4dae-86a9-dc801392d954","Type":"ContainerStarted","Data":"bfa908e32db75eef7e600ce051fdd42a976b60ebc4fec6749173d67383d58896"}
Nov 01 05:49:54.124723 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:54.124623    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7a360a-d4b7c96cf-cswr9"
Nov 01 05:49:54.238551 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:54.238501    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-7a360a-6cf998897-7v6d7" podStartSLOduration=2.23848791 podStartE2EDuration="2.23848791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:49:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:49:54.237709793 +0000 UTC m=+2772.308506507" watchObservedRunningTime="2026-11-01 05:49:54.23848791 +0000 UTC m=+2772.309284700"
Nov 01 05:49:55.193749 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:55.193717    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:49:57.638479 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:57.636497    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:49:57.812319 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:57.812286    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:49:57.873339 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:57.873284    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7a360a-d4b7c96cf-cswr9" podStartSLOduration=5.873268753 podStartE2EDuration="5.873268753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:49:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:49:54.383309214 +0000 UTC m=+2772.454105937" watchObservedRunningTime="2026-11-01 05:49:57.873268753 +0000 UTC m=+2775.944065474"
Nov 01 05:49:57.875159 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:57.875141    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:49:59.854324 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:49:59.854288    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:50:00.041804 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:50:00.041771    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:50:00.068674 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:50:00.068643    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:50:00.224417 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:50:00.224383    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:50:00.386143 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:50:00.386109    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-25565930-b8577d659-6pwlj"]
Nov 01 05:50:00.390378 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:50:00.390358    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-25565930-b8577d659-6pwlj"
Nov 01 05:50:00.412829 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:50:00.412797    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:50:00.483813 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:50:00.483725    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b679696-4dae-41d8-bc1f-1a598aeedec9-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-25565930-b8577d659-6pwlj\" (UID: \"5b679696-4dae-41d8-bc1f-1a598aeedec9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-25565930-b8577d659-6pwlj"
Nov 01 05:50:00.483995 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:50:00.483859    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g722\" (UniqueName: \"kubernetes.io/projected/5b679696-4dae-41d8-bc1f-1a598aeedec9-kube-api-access-6g722\") pod \"payload-pre-processing-e2e-mt-25565930-b8577d659-6pwlj\" (UID: \"5b679696-4dae-41d8-bc1f-1a598aeedec9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-25565930-b8577d659-6pwlj"
Nov 01 05:50:00.494222 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:50:00.494200    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-25565930-dockercfg-fxpr7\""
Nov 01 05:50:00.494222 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:50:00.494213    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-25565930\""
Nov 01 05:50:00.497846 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:50:00.497826    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-25565930-b8577d659-6pwlj"]
Nov 01 05:50:00.586075 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:50:00.586033    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g722\" (UniqueName: \"kubernetes.io/projected/5b679696-4dae-41d8-bc1f-1a598aeedec9-kube-api-access-6g722\") pod \"payload-pre-processing-e2e-mt-25565930-b8577d659-6pwlj\" (UID: \"5b679696-4dae-41d8-bc1f-1a598aeedec9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-25565930-b8577d659-6pwlj"
Nov 01 05:50:00.586296 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:50:00.586207    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b679696-4dae-41d8-bc1f-1a598aeedec9-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-25565930-b8577d659-6pwlj\" (UID: \"5b679696-4dae-41d8-bc1f-1a598aeedec9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-25565930-b8577d659-6pwlj"
Nov 01 05:50:00.587204 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:50:00.587158    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b679696-4dae-41d8-bc1f-1a598aeedec9-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-25565930-b8577d659-6pwlj\" (UID: \"5b679696-4dae-41d8-bc1f-1a598aeedec9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-25565930-b8577d659-6pwlj"
Nov 01 05:50:00.587821 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:50:00.587529    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-25565930-5c94467bfc-hp4rt"]
Nov 01 05:50:00.598605 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:50:00.598582    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-25565930-5c94467bfc-hp4rt"
Nov 01 05:50:00.607769 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:50:00.607738    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-25565930-5c94467bfc-hp4rt"]
Nov 01 05:50:00.658499 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:50:00.657738    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g722\" (UniqueName: \"kubernetes.io/projected/5b679696-4dae-41d8-bc1f-1a598aeedec9-kube-api-access-6g722\") pod \"payload-pre-processing-e2e-mt-25565930-b8577d659-6pwlj\" (UID: \"5b679696-4dae-41d8-bc1f-1a598aeedec9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-25565930-b8577d659-6pwlj"
Nov 01 05:50:00.686973 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:50:00.686947    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrhqh\" (UniqueName: \"kubernetes.io/projected/0e6b057a-6ebd-4a97-8ef9-e2c77f0379c0-kube-api-access-mrhqh\") pod \"payload-processing-e2e-mt-25565930-5c94467bfc-hp4rt\" (UID: \"0e6b057a-6ebd-4a97-8ef9-e2c77f0379c0\") " pod="openshift-ingress/payload-processing-e2e-mt-25565930-5c94467bfc-hp4rt"
Nov 01 05:50:00.687120 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:50:00.687093    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e6b057a-6ebd-4a97-8ef9-e2c77f0379c0-plugins-config-volume\") pod \"payload-processing-e2e-mt-25565930-5c94467bfc-hp4rt\" (UID: \"0e6b057a-6ebd-4a97-8ef9-e2c77f0379c0\") " pod="openshift-ingress/payload-processing-e2e-mt-25565930-5c94467bfc-hp4rt"
Nov 01 05:50:00.705960 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:50:00.705928    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-25565930-b8577d659-6pwlj"
Nov 01 05:50:00.788462 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:50:00.788414    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e6b057a-6ebd-4a97-8ef9-e2c77f0379c0-plugins-config-volume\") pod \"payload-processing-e2e-mt-25565930-5c94467bfc-hp4rt\" (UID: \"0e6b057a-6ebd-4a97-8ef9-e2c77f0379c0\") " pod="openshift-ingress/payload-processing-e2e-mt-25565930-5c94467bfc-hp4rt"
Nov 01 05:50:00.788647 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:50:00.788631    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrhqh\" (UniqueName: \"kubernetes.io/projected/0e6b057a-6ebd-4a97-8ef9-e2c77f0379c0-kube-api-access-mrhqh\") pod \"payload-processing-e2e-mt-25565930-5c94467bfc-hp4rt\" (UID: \"0e6b057a-6ebd-4a97-8ef9-e2c77f0379c0\") " pod="openshift-ingress/payload-processing-e2e-mt-25565930-5c94467bfc-hp4rt"
Nov 01 05:50:00.789128 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:50:00.789106    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e6b057a-6ebd-4a97-8ef9-e2c77f0379c0-plugins-config-volume\") pod \"payload-processing-e2e-mt-25565930-5c94467bfc-hp4rt\" (UID: \"0e6b057a-6ebd-4a97-8ef9-e2c77f0379c0\") " pod="openshift-ingress/payload-processing-e2e-mt-25565930-5c94467bfc-hp4rt"
Nov 01 05:50:00.849347 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:50:00.849313    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrhqh\" (UniqueName: \"kubernetes.io/projected/0e6b057a-6ebd-4a97-8ef9-e2c77f0379c0-kube-api-access-mrhqh\") pod \"payload-processing-e2e-mt-25565930-5c94467bfc-hp4rt\" (UID: \"0e6b057a-6ebd-4a97-8ef9-e2c77f0379c0\") " pod="openshift-ingress/payload-processing-e2e-mt-25565930-5c94467bfc-hp4rt"
Nov 01 05:50:00.974078 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:50:00.974048    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-25565930-5c94467bfc-hp4rt"
Nov 01 05:50:01.083893 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:50:01.083862    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-25565930-b8577d659-6pwlj"]
Nov 01 05:50:01.193850 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:50:01.193824    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-25565930-b8577d659-6pwlj" event={"ID":"5b679696-4dae-41d8-bc1f-1a598aeedec9","Type":"ContainerStarted","Data":"ec727af83bc419dc7e4c91e383b23efef072d4a143e5141fa6176d7c46f69120"}
Nov 01 05:50:01.193972 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:50:01.193877    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-25565930-b8577d659-6pwlj" event={"ID":"5b679696-4dae-41d8-bc1f-1a598aeedec9","Type":"ContainerStarted","Data":"f31f54621fc77069ede8dd2ed5f45076b76871f5b56ae67a16db19f2525f8956"}
Nov 01 05:50:01.194057 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:50:01.194031    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-25565930-b8577d659-6pwlj"
Nov 01 05:50:01.310891 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:50:01.310802    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-25565930-b8577d659-6pwlj" podStartSLOduration=1.3107888939999999 podStartE2EDuration="1.310788894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:50:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:50:01.308602616 +0000 UTC m=+2779.379399337" watchObservedRunningTime="2026-11-01 05:50:01.310788894 +0000 UTC m=+2779.381585613"
Nov 01 05:50:01.352789 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:50:01.352765    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-25565930-5c94467bfc-hp4rt"]
Nov 01 05:50:01.355268 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:50:01.355242    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e6b057a_6ebd_4a97_8ef9_e2c77f0379c0.slice/crio-15665e7b1da0fb856ddaee028d6136b45b157d21dd82ffc0fedd85260ad2a889 WatchSource:0}: Error finding container 15665e7b1da0fb856ddaee028d6136b45b157d21dd82ffc0fedd85260ad2a889: Status 404 returned error can't find the container with id 15665e7b1da0fb856ddaee028d6136b45b157d21dd82ffc0fedd85260ad2a889
Nov 01 05:50:02.200938 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:50:02.200904    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-25565930-5c94467bfc-hp4rt" event={"ID":"0e6b057a-6ebd-4a97-8ef9-e2c77f0379c0","Type":"ContainerStarted","Data":"1b75994dd0b73c7f3d7021be0a0c7bd45a9a0763a4f283c005ee0a020a712a35"}
Nov 01 05:50:02.201478 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:50:02.200969    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-25565930-5c94467bfc-hp4rt" event={"ID":"0e6b057a-6ebd-4a97-8ef9-e2c77f0379c0","Type":"ContainerStarted","Data":"15665e7b1da0fb856ddaee028d6136b45b157d21dd82ffc0fedd85260ad2a889"}
Nov 01 05:50:02.201478 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:50:02.201233    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-25565930-5c94467bfc-hp4rt"
Nov 01 05:50:02.318324 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:50:02.318273    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-25565930-5c94467bfc-hp4rt" podStartSLOduration=2.318257151 podStartE2EDuration="2.318257151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:50:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:50:02.317062878 +0000 UTC m=+2780.387859622" watchObservedRunningTime="2026-11-01 05:50:02.318257151 +0000 UTC m=+2780.389053871"
Nov 01 05:50:05.120069 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:50:05.120038    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:50:05.180900 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:50:05.180868    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7a360a-d4b7c96cf-cswr9"
Nov 01 05:50:05.182237 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:50:05.182217    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-7a360a-6cf998897-7v6d7"
Nov 01 05:50:05.328929 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:50:05.328900    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:50:09.468751 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:50:09.468718    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:50:09.914336 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:50:09.914306    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:50:12.204874 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:50:12.204846    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-25565930-b8577d659-6pwlj"
Nov 01 05:50:13.211363 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:50:13.211333    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-25565930-5c94467bfc-hp4rt"
Nov 01 05:50:14.710725 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:50:14.710691    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:50:19.468468 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:50:19.467331    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:50:20.984741 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:50:20.984663    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:50:24.290188 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:50:24.290150    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:50:28.996549 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:50:28.996516    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:50:30.068056 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:50:30.068023    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:50:33.810496 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:50:33.810430    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:50:38.601067 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:50:38.601035    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:50:42.378675 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:50:42.378645    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:50:43.268852 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:50:43.268824    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:50:43.461586 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:50:43.461391    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:50:43.638673 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:50:43.638582    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:50:48.187721 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:50:48.187689    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:50:48.462610 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:50:48.462524    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:50:48.644515 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:50:48.644481    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:50:48.816028 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:50:48.815958    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:50:53.351035 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:50:53.350984    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:50:53.535322 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:50:53.535286    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:50:53.711147 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:50:53.711116    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:50:57.661014 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:50:57.660978    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:50:59.392712 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:50:59.392675    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:00.069339 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:51:00.069306    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:51:03.133938 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:03.133907    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:03.249903 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:03.249867    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:51:03.320572 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:03.320541    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:04.199670 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:04.199638    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:04.720851 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:04.720818    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:05.236632 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:05.236599    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:05.829179 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:05.829148    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:06.484145 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:06.484110    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:06.708226 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:06.708157    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:09.348131 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:09.348100    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:09.825284 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:09.825238    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:10.475325 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:10.475291    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:10.762780 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:10.762705    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:10.837706 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:10.837677    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gbhn9"]
Nov 01 05:51:10.841717 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:10.841695    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gbhn9"
Nov 01 05:51:10.853427 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:10.853405    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e67e1922-ea2b-4fbc-858b-1648ec6124e9-istio-podinfo\") pod \"e2e-mt-25565930-openshift-default-64f5666649-gbhn9\" (UID: \"e67e1922-ea2b-4fbc-858b-1648ec6124e9\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gbhn9"
Nov 01 05:51:10.853565 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:10.853496    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e67e1922-ea2b-4fbc-858b-1648ec6124e9-istio-data\") pod \"e2e-mt-25565930-openshift-default-64f5666649-gbhn9\" (UID: \"e67e1922-ea2b-4fbc-858b-1648ec6124e9\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gbhn9"
Nov 01 05:51:10.853622 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:10.853560    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e67e1922-ea2b-4fbc-858b-1648ec6124e9-istio-envoy\") pod \"e2e-mt-25565930-openshift-default-64f5666649-gbhn9\" (UID: \"e67e1922-ea2b-4fbc-858b-1648ec6124e9\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gbhn9"
Nov 01 05:51:10.853622 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:10.853605    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e67e1922-ea2b-4fbc-858b-1648ec6124e9-istiod-ca-cert\") pod \"e2e-mt-25565930-openshift-default-64f5666649-gbhn9\" (UID: \"e67e1922-ea2b-4fbc-858b-1648ec6124e9\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gbhn9"
Nov 01 05:51:10.853707 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:10.853676    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e67e1922-ea2b-4fbc-858b-1648ec6124e9-workload-socket\") pod \"e2e-mt-25565930-openshift-default-64f5666649-gbhn9\" (UID: \"e67e1922-ea2b-4fbc-858b-1648ec6124e9\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gbhn9"
Nov 01 05:51:10.853707 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:10.853696    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e67e1922-ea2b-4fbc-858b-1648ec6124e9-istio-token\") pod \"e2e-mt-25565930-openshift-default-64f5666649-gbhn9\" (UID: \"e67e1922-ea2b-4fbc-858b-1648ec6124e9\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gbhn9"
Nov 01 05:51:10.853800 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:10.853727    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pv82\" (UniqueName: \"kubernetes.io/projected/e67e1922-ea2b-4fbc-858b-1648ec6124e9-kube-api-access-5pv82\") pod \"e2e-mt-25565930-openshift-default-64f5666649-gbhn9\" (UID: \"e67e1922-ea2b-4fbc-858b-1648ec6124e9\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gbhn9"
Nov 01 05:51:10.854012 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:10.853994    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e67e1922-ea2b-4fbc-858b-1648ec6124e9-credential-socket\") pod \"e2e-mt-25565930-openshift-default-64f5666649-gbhn9\" (UID: \"e67e1922-ea2b-4fbc-858b-1648ec6124e9\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gbhn9"
Nov 01 05:51:10.854065 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:10.854042    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e67e1922-ea2b-4fbc-858b-1648ec6124e9-workload-certs\") pod \"e2e-mt-25565930-openshift-default-64f5666649-gbhn9\" (UID: \"e67e1922-ea2b-4fbc-858b-1648ec6124e9\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gbhn9"
Nov 01 05:51:10.893860 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:10.893834    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-25565930-openshift-default-dockercfg-f64wx\""
Nov 01 05:51:10.946153 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:10.946127    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gbhn9"]
Nov 01 05:51:10.955007 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:10.954982    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e67e1922-ea2b-4fbc-858b-1648ec6124e9-credential-socket\") pod \"e2e-mt-25565930-openshift-default-64f5666649-gbhn9\" (UID: \"e67e1922-ea2b-4fbc-858b-1648ec6124e9\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gbhn9"
Nov 01 05:51:10.955131 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:10.955037    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e67e1922-ea2b-4fbc-858b-1648ec6124e9-workload-certs\") pod \"e2e-mt-25565930-openshift-default-64f5666649-gbhn9\" (UID: \"e67e1922-ea2b-4fbc-858b-1648ec6124e9\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gbhn9"
Nov 01 05:51:10.955131 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:10.955081    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e67e1922-ea2b-4fbc-858b-1648ec6124e9-istio-podinfo\") pod \"e2e-mt-25565930-openshift-default-64f5666649-gbhn9\" (UID: \"e67e1922-ea2b-4fbc-858b-1648ec6124e9\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gbhn9"
Nov 01 05:51:10.955272 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:10.955141    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e67e1922-ea2b-4fbc-858b-1648ec6124e9-istio-data\") pod \"e2e-mt-25565930-openshift-default-64f5666649-gbhn9\" (UID: \"e67e1922-ea2b-4fbc-858b-1648ec6124e9\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gbhn9"
Nov 01 05:51:10.955272 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:10.955168    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e67e1922-ea2b-4fbc-858b-1648ec6124e9-istio-envoy\") pod \"e2e-mt-25565930-openshift-default-64f5666649-gbhn9\" (UID: \"e67e1922-ea2b-4fbc-858b-1648ec6124e9\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gbhn9"
Nov 01 05:51:10.955272 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:10.955207    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e67e1922-ea2b-4fbc-858b-1648ec6124e9-istiod-ca-cert\") pod \"e2e-mt-25565930-openshift-default-64f5666649-gbhn9\" (UID: \"e67e1922-ea2b-4fbc-858b-1648ec6124e9\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gbhn9"
Nov 01 05:51:10.955655 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:10.955275    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e67e1922-ea2b-4fbc-858b-1648ec6124e9-workload-socket\") pod \"e2e-mt-25565930-openshift-default-64f5666649-gbhn9\" (UID: \"e67e1922-ea2b-4fbc-858b-1648ec6124e9\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gbhn9"
Nov 01 05:51:10.955655 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:10.955306    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e67e1922-ea2b-4fbc-858b-1648ec6124e9-istio-token\") pod \"e2e-mt-25565930-openshift-default-64f5666649-gbhn9\" (UID: \"e67e1922-ea2b-4fbc-858b-1648ec6124e9\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gbhn9"
Nov 01 05:51:10.955655 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:10.955366    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pv82\" (UniqueName: \"kubernetes.io/projected/e67e1922-ea2b-4fbc-858b-1648ec6124e9-kube-api-access-5pv82\") pod \"e2e-mt-25565930-openshift-default-64f5666649-gbhn9\" (UID: \"e67e1922-ea2b-4fbc-858b-1648ec6124e9\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gbhn9"
Nov 01 05:51:10.955655 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:10.955416    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e67e1922-ea2b-4fbc-858b-1648ec6124e9-credential-socket\") pod \"e2e-mt-25565930-openshift-default-64f5666649-gbhn9\" (UID: \"e67e1922-ea2b-4fbc-858b-1648ec6124e9\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gbhn9"
Nov 01 05:51:10.956126 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:10.956099    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e67e1922-ea2b-4fbc-858b-1648ec6124e9-istiod-ca-cert\") pod \"e2e-mt-25565930-openshift-default-64f5666649-gbhn9\" (UID: \"e67e1922-ea2b-4fbc-858b-1648ec6124e9\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gbhn9"
Nov 01 05:51:10.958808 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:10.958760    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e67e1922-ea2b-4fbc-858b-1648ec6124e9-istio-data\") pod \"e2e-mt-25565930-openshift-default-64f5666649-gbhn9\" (UID: \"e67e1922-ea2b-4fbc-858b-1648ec6124e9\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gbhn9"
Nov 01 05:51:10.959188 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:10.959162    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e67e1922-ea2b-4fbc-858b-1648ec6124e9-workload-socket\") pod \"e2e-mt-25565930-openshift-default-64f5666649-gbhn9\" (UID: \"e67e1922-ea2b-4fbc-858b-1648ec6124e9\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gbhn9"
Nov 01 05:51:10.959724 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:10.959699    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e67e1922-ea2b-4fbc-858b-1648ec6124e9-workload-certs\") pod \"e2e-mt-25565930-openshift-default-64f5666649-gbhn9\" (UID: \"e67e1922-ea2b-4fbc-858b-1648ec6124e9\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gbhn9"
Nov 01 05:51:10.961825 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:10.961799    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e67e1922-ea2b-4fbc-858b-1648ec6124e9-istio-envoy\") pod \"e2e-mt-25565930-openshift-default-64f5666649-gbhn9\" (UID: \"e67e1922-ea2b-4fbc-858b-1648ec6124e9\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gbhn9"
Nov 01 05:51:10.962555 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:10.962505    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e67e1922-ea2b-4fbc-858b-1648ec6124e9-istio-podinfo\") pod \"e2e-mt-25565930-openshift-default-64f5666649-gbhn9\" (UID: \"e67e1922-ea2b-4fbc-858b-1648ec6124e9\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gbhn9"
Nov 01 05:51:10.977834 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:10.977809    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:11.017128 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:11.017066    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e67e1922-ea2b-4fbc-858b-1648ec6124e9-istio-token\") pod \"e2e-mt-25565930-openshift-default-64f5666649-gbhn9\" (UID: \"e67e1922-ea2b-4fbc-858b-1648ec6124e9\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gbhn9"
Nov 01 05:51:11.017251 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:11.017125    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pv82\" (UniqueName: \"kubernetes.io/projected/e67e1922-ea2b-4fbc-858b-1648ec6124e9-kube-api-access-5pv82\") pod \"e2e-mt-25565930-openshift-default-64f5666649-gbhn9\" (UID: \"e67e1922-ea2b-4fbc-858b-1648ec6124e9\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gbhn9"
Nov 01 05:51:11.155345 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:11.155320    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gbhn9"
Nov 01 05:51:11.593088 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:11.592926    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gbhn9"]
Nov 01 05:51:11.595739 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:51:11.595708    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode67e1922_ea2b_4fbc_858b_1648ec6124e9.slice/crio-4d16c36ae9966399f4ecb830e5f8cae16ea432d05d8e6667be57fd584b7641d0 WatchSource:0}: Error finding container 4d16c36ae9966399f4ecb830e5f8cae16ea432d05d8e6667be57fd584b7641d0: Status 404 returned error can't find the container with id 4d16c36ae9966399f4ecb830e5f8cae16ea432d05d8e6667be57fd584b7641d0
Nov 01 05:51:11.598165 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:11.598135    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:51:11.598238 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:11.598202    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:51:11.598274 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:11.598234    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:51:11.696873 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:11.696840    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gbhn9" event={"ID":"e67e1922-ea2b-4fbc-858b-1648ec6124e9","Type":"ContainerStarted","Data":"7ae3b46ccabc42915dfdc51dafc01429fa1d2b7fb7e917fe70904e176d03fb08"}
Nov 01 05:51:11.697064 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:11.696892    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gbhn9" event={"ID":"e67e1922-ea2b-4fbc-858b-1648ec6124e9","Type":"ContainerStarted","Data":"4d16c36ae9966399f4ecb830e5f8cae16ea432d05d8e6667be57fd584b7641d0"}
Nov 01 05:51:11.817720 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:11.817667    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gbhn9" podStartSLOduration=1.817652915 podStartE2EDuration="1.817652915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:51:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:51:11.81571385 +0000 UTC m=+2849.886510572" watchObservedRunningTime="2026-11-01 05:51:11.817652915 +0000 UTC m=+2849.888449635"
Nov 01 05:51:12.155780 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:12.155753    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gbhn9"
Nov 01 05:51:12.157223 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:12.157199    5689 patch_prober.go:29] interesting pod/e2e-mt-25565930-openshift-default-64f5666649-gbhn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Nov 01 05:51:12.157285 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:12.157253    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gbhn9" podUID="e67e1922-ea2b-4fbc-858b-1648ec6124e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Nov 01 05:51:13.156427 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:13.156379    5689 patch_prober.go:29] interesting pod/e2e-mt-25565930-openshift-default-64f5666649-gbhn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Nov 01 05:51:13.156821 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:13.156442    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gbhn9" podUID="e67e1922-ea2b-4fbc-858b-1648ec6124e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Nov 01 05:51:13.762000 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:13.761966    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:13.951227 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:13.951198    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:14.156238 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:14.156204    5689 patch_prober.go:29] interesting pod/e2e-mt-25565930-openshift-default-64f5666649-gbhn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Nov 01 05:51:14.156418 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:14.156258    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gbhn9" podUID="e67e1922-ea2b-4fbc-858b-1648ec6124e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Nov 01 05:51:14.354477 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:14.347381    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc"]
Nov 01 05:51:14.380529 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:14.380498    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc"
Nov 01 05:51:14.443002 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:14.442959    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b2091c5-03e9-45a0-94d3-3946bcd2db7a-workload-certs\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc\" (UID: \"5b2091c5-03e9-45a0-94d3-3946bcd2db7a\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc"
Nov 01 05:51:14.443170 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:14.443011    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b2091c5-03e9-45a0-94d3-3946bcd2db7a-istio-data\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc\" (UID: \"5b2091c5-03e9-45a0-94d3-3946bcd2db7a\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc"
Nov 01 05:51:14.443170 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:14.443051    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b2091c5-03e9-45a0-94d3-3946bcd2db7a-workload-socket\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc\" (UID: \"5b2091c5-03e9-45a0-94d3-3946bcd2db7a\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc"
Nov 01 05:51:14.443170 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:14.443097    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b2091c5-03e9-45a0-94d3-3946bcd2db7a-credential-socket\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc\" (UID: \"5b2091c5-03e9-45a0-94d3-3946bcd2db7a\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc"
Nov 01 05:51:14.443170 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:14.443156    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b2091c5-03e9-45a0-94d3-3946bcd2db7a-istio-podinfo\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc\" (UID: \"5b2091c5-03e9-45a0-94d3-3946bcd2db7a\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc"
Nov 01 05:51:14.443394 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:14.443228    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b2091c5-03e9-45a0-94d3-3946bcd2db7a-istiod-ca-cert\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc\" (UID: \"5b2091c5-03e9-45a0-94d3-3946bcd2db7a\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc"
Nov 01 05:51:14.443394 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:14.443315    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b2091c5-03e9-45a0-94d3-3946bcd2db7a-istio-envoy\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc\" (UID: \"5b2091c5-03e9-45a0-94d3-3946bcd2db7a\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc"
Nov 01 05:51:14.443532 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:14.443442    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl42b\" (UniqueName: \"kubernetes.io/projected/5b2091c5-03e9-45a0-94d3-3946bcd2db7a-kube-api-access-fl42b\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc\" (UID: \"5b2091c5-03e9-45a0-94d3-3946bcd2db7a\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc"
Nov 01 05:51:14.443532 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:14.443484    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b2091c5-03e9-45a0-94d3-3946bcd2db7a-istio-token\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc\" (UID: \"5b2091c5-03e9-45a0-94d3-3946bcd2db7a\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc"
Nov 01 05:51:14.447016 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:14.446807    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc"]
Nov 01 05:51:14.465416 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:14.465382    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-7a360a-openshift-default-dockercfg-ggnz9\""
Nov 01 05:51:14.544266 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:14.544235    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b2091c5-03e9-45a0-94d3-3946bcd2db7a-credential-socket\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc\" (UID: \"5b2091c5-03e9-45a0-94d3-3946bcd2db7a\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc"
Nov 01 05:51:14.544514 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:14.544289    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b2091c5-03e9-45a0-94d3-3946bcd2db7a-istio-podinfo\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc\" (UID: \"5b2091c5-03e9-45a0-94d3-3946bcd2db7a\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc"
Nov 01 05:51:14.544514 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:14.544329    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b2091c5-03e9-45a0-94d3-3946bcd2db7a-istiod-ca-cert\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc\" (UID: \"5b2091c5-03e9-45a0-94d3-3946bcd2db7a\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc"
Nov 01 05:51:14.544514 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:14.544350    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b2091c5-03e9-45a0-94d3-3946bcd2db7a-istio-envoy\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc\" (UID: \"5b2091c5-03e9-45a0-94d3-3946bcd2db7a\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc"
Nov 01 05:51:14.544514 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:14.544494    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl42b\" (UniqueName: \"kubernetes.io/projected/5b2091c5-03e9-45a0-94d3-3946bcd2db7a-kube-api-access-fl42b\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc\" (UID: \"5b2091c5-03e9-45a0-94d3-3946bcd2db7a\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc"
Nov 01 05:51:14.544780 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:14.544552    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b2091c5-03e9-45a0-94d3-3946bcd2db7a-istio-token\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc\" (UID: \"5b2091c5-03e9-45a0-94d3-3946bcd2db7a\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc"
Nov 01 05:51:14.544780 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:14.544629    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b2091c5-03e9-45a0-94d3-3946bcd2db7a-workload-certs\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc\" (UID: \"5b2091c5-03e9-45a0-94d3-3946bcd2db7a\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc"
Nov 01 05:51:14.544780 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:14.544663    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b2091c5-03e9-45a0-94d3-3946bcd2db7a-credential-socket\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc\" (UID: \"5b2091c5-03e9-45a0-94d3-3946bcd2db7a\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc"
Nov 01 05:51:14.544780 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:14.544678    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b2091c5-03e9-45a0-94d3-3946bcd2db7a-istio-data\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc\" (UID: \"5b2091c5-03e9-45a0-94d3-3946bcd2db7a\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc"
Nov 01 05:51:14.544780 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:14.544753    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b2091c5-03e9-45a0-94d3-3946bcd2db7a-workload-socket\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc\" (UID: \"5b2091c5-03e9-45a0-94d3-3946bcd2db7a\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc"
Nov 01 05:51:14.544967 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:14.544908    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b2091c5-03e9-45a0-94d3-3946bcd2db7a-workload-certs\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc\" (UID: \"5b2091c5-03e9-45a0-94d3-3946bcd2db7a\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc"
Nov 01 05:51:14.545149 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:14.545124    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b2091c5-03e9-45a0-94d3-3946bcd2db7a-istio-data\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc\" (UID: \"5b2091c5-03e9-45a0-94d3-3946bcd2db7a\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc"
Nov 01 05:51:14.545217 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:14.545150    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b2091c5-03e9-45a0-94d3-3946bcd2db7a-workload-socket\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc\" (UID: \"5b2091c5-03e9-45a0-94d3-3946bcd2db7a\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc"
Nov 01 05:51:14.545217 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:14.545162    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b2091c5-03e9-45a0-94d3-3946bcd2db7a-istiod-ca-cert\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc\" (UID: \"5b2091c5-03e9-45a0-94d3-3946bcd2db7a\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc"
Nov 01 05:51:14.546963 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:14.546941    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b2091c5-03e9-45a0-94d3-3946bcd2db7a-istio-envoy\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc\" (UID: \"5b2091c5-03e9-45a0-94d3-3946bcd2db7a\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc"
Nov 01 05:51:14.547205 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:14.547175    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b2091c5-03e9-45a0-94d3-3946bcd2db7a-istio-podinfo\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc\" (UID: \"5b2091c5-03e9-45a0-94d3-3946bcd2db7a\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc"
Nov 01 05:51:14.605485 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:14.604109    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:14.605485 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:14.605282    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b2091c5-03e9-45a0-94d3-3946bcd2db7a-istio-token\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc\" (UID: \"5b2091c5-03e9-45a0-94d3-3946bcd2db7a\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc"
Nov 01 05:51:14.605706 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:14.605680    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl42b\" (UniqueName: \"kubernetes.io/projected/5b2091c5-03e9-45a0-94d3-3946bcd2db7a-kube-api-access-fl42b\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc\" (UID: \"5b2091c5-03e9-45a0-94d3-3946bcd2db7a\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc"
Nov 01 05:51:14.752741 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:14.752647    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc"
Nov 01 05:51:15.156504 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:15.156442    5689 patch_prober.go:29] interesting pod/e2e-mt-25565930-openshift-default-64f5666649-gbhn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Nov 01 05:51:15.156688 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:15.156518    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gbhn9" podUID="e67e1922-ea2b-4fbc-858b-1648ec6124e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Nov 01 05:51:15.219508 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:15.219358    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc"]
Nov 01 05:51:15.222719 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:51:15.222675    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b2091c5_03e9_45a0_94d3_3946bcd2db7a.slice/crio-0ff7796c97ba9bb5b00a012a0625cf43c6ed2461170b12424baa77bf66f6a424 WatchSource:0}: Error finding container 0ff7796c97ba9bb5b00a012a0625cf43c6ed2461170b12424baa77bf66f6a424: Status 404 returned error can't find the container with id 0ff7796c97ba9bb5b00a012a0625cf43c6ed2461170b12424baa77bf66f6a424
Nov 01 05:51:15.225702 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:15.225657    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:51:15.225828 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:15.225753    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:51:15.225828 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:15.225802    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:51:15.632320 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:15.632245    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:15.728882 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:15.728829    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc" event={"ID":"5b2091c5-03e9-45a0-94d3-3946bcd2db7a","Type":"ContainerStarted","Data":"0a138be10c9fc7d89093b455c384b90188907d0c73769aee5b0ea14e73a1e7e9"}
Nov 01 05:51:15.728882 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:15.728879    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc" event={"ID":"5b2091c5-03e9-45a0-94d3-3946bcd2db7a","Type":"ContainerStarted","Data":"0ff7796c97ba9bb5b00a012a0625cf43c6ed2461170b12424baa77bf66f6a424"}
Nov 01 05:51:15.752951 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:15.752713    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc"
Nov 01 05:51:15.816381 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:15.816337    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:15.853150 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:15.853063    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc" podStartSLOduration=1.853045758 podStartE2EDuration="1.853045758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:51:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:51:15.850872267 +0000 UTC m=+2853.921669022" watchObservedRunningTime="2026-11-01 05:51:15.853045758 +0000 UTC m=+2853.923842480"
Nov 01 05:51:16.170476 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:16.165718    5689 patch_prober.go:29] interesting pod/e2e-mt-25565930-openshift-default-64f5666649-gbhn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Nov 01 05:51:16.170476 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:16.165776    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gbhn9" podUID="e67e1922-ea2b-4fbc-858b-1648ec6124e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Nov 01 05:51:16.754639 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:16.754608    5689 patch_prober.go:29] interesting pod/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body=
Nov 01 05:51:16.755438 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:16.755416    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc" podUID="5b2091c5-03e9-45a0-94d3-3946bcd2db7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused"
Nov 01 05:51:17.155767 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:17.155735    5689 patch_prober.go:29] interesting pod/e2e-mt-25565930-openshift-default-64f5666649-gbhn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Nov 01 05:51:17.155930 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:17.155793    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gbhn9" podUID="e67e1922-ea2b-4fbc-858b-1648ec6124e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Nov 01 05:51:17.753513 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:17.753479    5689 patch_prober.go:29] interesting pod/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body=
Nov 01 05:51:17.753689 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:17.753532    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc" podUID="5b2091c5-03e9-45a0-94d3-3946bcd2db7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused"
Nov 01 05:51:18.156363 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:18.156325    5689 patch_prober.go:29] interesting pod/e2e-mt-25565930-openshift-default-64f5666649-gbhn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Nov 01 05:51:18.156894 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:18.156388    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gbhn9" podUID="e67e1922-ea2b-4fbc-858b-1648ec6124e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Nov 01 05:51:18.752901 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:18.752870    5689 patch_prober.go:29] interesting pod/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body=
Nov 01 05:51:18.753091 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:18.752925    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc" podUID="5b2091c5-03e9-45a0-94d3-3946bcd2db7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused"
Nov 01 05:51:19.160362 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:19.160335    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gbhn9"
Nov 01 05:51:19.160775 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:19.160755    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gbhn9"
Nov 01 05:51:19.161508 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:19.161487    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gbhn9"
Nov 01 05:51:19.753609 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:19.753579    5689 patch_prober.go:29] interesting pod/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body=
Nov 01 05:51:19.753800 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:19.753649    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc" podUID="5b2091c5-03e9-45a0-94d3-3946bcd2db7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused"
Nov 01 05:51:20.140277 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:20.140196    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:20.753359 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:20.753326    5689 patch_prober.go:29] interesting pod/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body=
Nov 01 05:51:20.753780 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:20.753372    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc" podUID="5b2091c5-03e9-45a0-94d3-3946bcd2db7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused"
Nov 01 05:51:21.758019 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:21.757986    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc"
Nov 01 05:51:21.758490 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:21.758369    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc"
Nov 01 05:51:21.759131 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:21.759109    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc"
Nov 01 05:51:22.548612 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:22.548579    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:22.752764 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:22.739403    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:24.912105 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:24.912064    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:25.973000 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:25.972966    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-cpzlj"]
Nov 01 05:51:25.976910 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:25.976890    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-cpzlj"
Nov 01 05:51:26.129677 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:26.129644    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75700ead-89a9-4ee7-a12a-934b6c5cb2cb-credential-socket\") pod \"e2e-mt-25565930-openshift-default-64f5666649-cpzlj\" (UID: \"75700ead-89a9-4ee7-a12a-934b6c5cb2cb\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-cpzlj"
Nov 01 05:51:26.129845 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:26.129762    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75700ead-89a9-4ee7-a12a-934b6c5cb2cb-istio-data\") pod \"e2e-mt-25565930-openshift-default-64f5666649-cpzlj\" (UID: \"75700ead-89a9-4ee7-a12a-934b6c5cb2cb\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-cpzlj"
Nov 01 05:51:26.129892 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:26.129841    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75700ead-89a9-4ee7-a12a-934b6c5cb2cb-istio-podinfo\") pod \"e2e-mt-25565930-openshift-default-64f5666649-cpzlj\" (UID: \"75700ead-89a9-4ee7-a12a-934b6c5cb2cb\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-cpzlj"
Nov 01 05:51:26.129892 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:26.129860    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75700ead-89a9-4ee7-a12a-934b6c5cb2cb-istio-token\") pod \"e2e-mt-25565930-openshift-default-64f5666649-cpzlj\" (UID: \"75700ead-89a9-4ee7-a12a-934b6c5cb2cb\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-cpzlj"
Nov 01 05:51:26.129963 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:26.129953    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75700ead-89a9-4ee7-a12a-934b6c5cb2cb-istio-envoy\") pod \"e2e-mt-25565930-openshift-default-64f5666649-cpzlj\" (UID: \"75700ead-89a9-4ee7-a12a-934b6c5cb2cb\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-cpzlj"
Nov 01 05:51:26.130045 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:26.130030    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75700ead-89a9-4ee7-a12a-934b6c5cb2cb-workload-socket\") pod \"e2e-mt-25565930-openshift-default-64f5666649-cpzlj\" (UID: \"75700ead-89a9-4ee7-a12a-934b6c5cb2cb\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-cpzlj"
Nov 01 05:51:26.130091 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:26.130078    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75700ead-89a9-4ee7-a12a-934b6c5cb2cb-workload-certs\") pod \"e2e-mt-25565930-openshift-default-64f5666649-cpzlj\" (UID: \"75700ead-89a9-4ee7-a12a-934b6c5cb2cb\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-cpzlj"
Nov 01 05:51:26.130128 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:26.130099    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75700ead-89a9-4ee7-a12a-934b6c5cb2cb-istiod-ca-cert\") pod \"e2e-mt-25565930-openshift-default-64f5666649-cpzlj\" (UID: \"75700ead-89a9-4ee7-a12a-934b6c5cb2cb\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-cpzlj"
Nov 01 05:51:26.130128 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:26.130114    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdshb\" (UniqueName: \"kubernetes.io/projected/75700ead-89a9-4ee7-a12a-934b6c5cb2cb-kube-api-access-cdshb\") pod \"e2e-mt-25565930-openshift-default-64f5666649-cpzlj\" (UID: \"75700ead-89a9-4ee7-a12a-934b6c5cb2cb\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-cpzlj"
Nov 01 05:51:26.181282 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:26.181249    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-cpzlj"]
Nov 01 05:51:26.231726 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:26.231642    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75700ead-89a9-4ee7-a12a-934b6c5cb2cb-istio-data\") pod \"e2e-mt-25565930-openshift-default-64f5666649-cpzlj\" (UID: \"75700ead-89a9-4ee7-a12a-934b6c5cb2cb\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-cpzlj"
Nov 01 05:51:26.231890 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:26.231749    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75700ead-89a9-4ee7-a12a-934b6c5cb2cb-istio-podinfo\") pod \"e2e-mt-25565930-openshift-default-64f5666649-cpzlj\" (UID: \"75700ead-89a9-4ee7-a12a-934b6c5cb2cb\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-cpzlj"
Nov 01 05:51:26.231890 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:26.231775    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75700ead-89a9-4ee7-a12a-934b6c5cb2cb-istio-token\") pod \"e2e-mt-25565930-openshift-default-64f5666649-cpzlj\" (UID: \"75700ead-89a9-4ee7-a12a-934b6c5cb2cb\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-cpzlj"
Nov 01 05:51:26.231890 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:26.231812    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75700ead-89a9-4ee7-a12a-934b6c5cb2cb-istio-envoy\") pod \"e2e-mt-25565930-openshift-default-64f5666649-cpzlj\" (UID: \"75700ead-89a9-4ee7-a12a-934b6c5cb2cb\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-cpzlj"
Nov 01 05:51:26.232079 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:26.231906    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75700ead-89a9-4ee7-a12a-934b6c5cb2cb-workload-socket\") pod \"e2e-mt-25565930-openshift-default-64f5666649-cpzlj\" (UID: \"75700ead-89a9-4ee7-a12a-934b6c5cb2cb\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-cpzlj"
Nov 01 05:51:26.232079 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:26.231944    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75700ead-89a9-4ee7-a12a-934b6c5cb2cb-workload-certs\") pod \"e2e-mt-25565930-openshift-default-64f5666649-cpzlj\" (UID: \"75700ead-89a9-4ee7-a12a-934b6c5cb2cb\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-cpzlj"
Nov 01 05:51:26.232079 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:26.231968    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75700ead-89a9-4ee7-a12a-934b6c5cb2cb-istiod-ca-cert\") pod \"e2e-mt-25565930-openshift-default-64f5666649-cpzlj\" (UID: \"75700ead-89a9-4ee7-a12a-934b6c5cb2cb\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-cpzlj"
Nov 01 05:51:26.232079 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:26.231994    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdshb\" (UniqueName: \"kubernetes.io/projected/75700ead-89a9-4ee7-a12a-934b6c5cb2cb-kube-api-access-cdshb\") pod \"e2e-mt-25565930-openshift-default-64f5666649-cpzlj\" (UID: \"75700ead-89a9-4ee7-a12a-934b6c5cb2cb\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-cpzlj"
Nov 01 05:51:26.232079 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:26.232050    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75700ead-89a9-4ee7-a12a-934b6c5cb2cb-credential-socket\") pod \"e2e-mt-25565930-openshift-default-64f5666649-cpzlj\" (UID: \"75700ead-89a9-4ee7-a12a-934b6c5cb2cb\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-cpzlj"
Nov 01 05:51:26.232351 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:26.232075    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75700ead-89a9-4ee7-a12a-934b6c5cb2cb-istio-data\") pod \"e2e-mt-25565930-openshift-default-64f5666649-cpzlj\" (UID: \"75700ead-89a9-4ee7-a12a-934b6c5cb2cb\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-cpzlj"
Nov 01 05:51:26.232351 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:26.232305    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75700ead-89a9-4ee7-a12a-934b6c5cb2cb-workload-socket\") pod \"e2e-mt-25565930-openshift-default-64f5666649-cpzlj\" (UID: \"75700ead-89a9-4ee7-a12a-934b6c5cb2cb\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-cpzlj"
Nov 01 05:51:26.232351 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:26.232326    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75700ead-89a9-4ee7-a12a-934b6c5cb2cb-workload-certs\") pod \"e2e-mt-25565930-openshift-default-64f5666649-cpzlj\" (UID: \"75700ead-89a9-4ee7-a12a-934b6c5cb2cb\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-cpzlj"
Nov 01 05:51:26.232551 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:26.232409    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75700ead-89a9-4ee7-a12a-934b6c5cb2cb-credential-socket\") pod \"e2e-mt-25565930-openshift-default-64f5666649-cpzlj\" (UID: \"75700ead-89a9-4ee7-a12a-934b6c5cb2cb\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-cpzlj"
Nov 01 05:51:26.232650 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:26.232634    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75700ead-89a9-4ee7-a12a-934b6c5cb2cb-istiod-ca-cert\") pod \"e2e-mt-25565930-openshift-default-64f5666649-cpzlj\" (UID: \"75700ead-89a9-4ee7-a12a-934b6c5cb2cb\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-cpzlj"
Nov 01 05:51:26.234292 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:26.234269    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75700ead-89a9-4ee7-a12a-934b6c5cb2cb-istio-envoy\") pod \"e2e-mt-25565930-openshift-default-64f5666649-cpzlj\" (UID: \"75700ead-89a9-4ee7-a12a-934b6c5cb2cb\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-cpzlj"
Nov 01 05:51:26.234439 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:26.234422    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75700ead-89a9-4ee7-a12a-934b6c5cb2cb-istio-podinfo\") pod \"e2e-mt-25565930-openshift-default-64f5666649-cpzlj\" (UID: \"75700ead-89a9-4ee7-a12a-934b6c5cb2cb\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-cpzlj"
Nov 01 05:51:26.292344 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:26.292312    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75700ead-89a9-4ee7-a12a-934b6c5cb2cb-istio-token\") pod \"e2e-mt-25565930-openshift-default-64f5666649-cpzlj\" (UID: \"75700ead-89a9-4ee7-a12a-934b6c5cb2cb\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-cpzlj"
Nov 01 05:51:26.292480 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:26.292427    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdshb\" (UniqueName: \"kubernetes.io/projected/75700ead-89a9-4ee7-a12a-934b6c5cb2cb-kube-api-access-cdshb\") pod \"e2e-mt-25565930-openshift-default-64f5666649-cpzlj\" (UID: \"75700ead-89a9-4ee7-a12a-934b6c5cb2cb\") " pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-cpzlj"
Nov 01 05:51:26.591513 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:26.591398    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-cpzlj"
Nov 01 05:51:27.021988 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:27.021841    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-cpzlj"]
Nov 01 05:51:27.024745 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:51:27.024714    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75700ead_89a9_4ee7_a12a_934b6c5cb2cb.slice/crio-6a9df6c0f28a02796c3f3644a2ebfd8eade450655625ad94538a8c6a779ab826 WatchSource:0}: Error finding container 6a9df6c0f28a02796c3f3644a2ebfd8eade450655625ad94538a8c6a779ab826: Status 404 returned error can't find the container with id 6a9df6c0f28a02796c3f3644a2ebfd8eade450655625ad94538a8c6a779ab826
Nov 01 05:51:27.027575 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:27.027545    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:51:27.027652 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:27.027610    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:51:27.027652 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:27.027637    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:51:27.809802 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:27.809765    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-cpzlj" event={"ID":"75700ead-89a9-4ee7-a12a-934b6c5cb2cb","Type":"ContainerStarted","Data":"535e7da13a60fc5686a3870e965895709a6a154b689ef823b37d7ebadca43938"}
Nov 01 05:51:27.809985 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:27.809832    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-cpzlj" event={"ID":"75700ead-89a9-4ee7-a12a-934b6c5cb2cb","Type":"ContainerStarted","Data":"6a9df6c0f28a02796c3f3644a2ebfd8eade450655625ad94538a8c6a779ab826"}
Nov 01 05:51:27.984427 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:27.984376    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-cpzlj" podStartSLOduration=2.984358879 podStartE2EDuration="2.984358879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:51:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:51:27.981745332 +0000 UTC m=+2866.052542066" watchObservedRunningTime="2026-11-01 05:51:27.984358879 +0000 UTC m=+2866.055155598"
Nov 01 05:51:28.594881 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:28.594802    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-cpzlj"
Nov 01 05:51:28.601133 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:28.601103    5689 patch_prober.go:29] interesting pod/e2e-mt-25565930-openshift-default-64f5666649-cpzlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body=
Nov 01 05:51:28.601321 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:28.601161    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-cpzlj" podUID="75700ead-89a9-4ee7-a12a-934b6c5cb2cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused"
Nov 01 05:51:29.595479 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:29.593910    5689 patch_prober.go:29] interesting pod/e2e-mt-25565930-openshift-default-64f5666649-cpzlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body=
Nov 01 05:51:29.595479 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:29.593965    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-cpzlj" podUID="75700ead-89a9-4ee7-a12a-934b6c5cb2cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused"
Nov 01 05:51:29.762146 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:29.762109    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:30.069710 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:51:30.069676    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:51:30.595854 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:30.595812    5689 patch_prober.go:29] interesting pod/e2e-mt-25565930-openshift-default-64f5666649-cpzlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body=
Nov 01 05:51:30.596439 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:30.596405    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-cpzlj" podUID="75700ead-89a9-4ee7-a12a-934b6c5cb2cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused"
Nov 01 05:51:31.591836 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:31.591797    5689 patch_prober.go:29] interesting pod/e2e-mt-25565930-openshift-default-64f5666649-cpzlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body=
Nov 01 05:51:31.592146 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:31.592115    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-cpzlj" podUID="75700ead-89a9-4ee7-a12a-934b6c5cb2cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused"
Nov 01 05:51:32.592259 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:32.592220    5689 patch_prober.go:29] interesting pod/e2e-mt-25565930-openshift-default-64f5666649-cpzlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body=
Nov 01 05:51:32.592871 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:32.592841    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-cpzlj" podUID="75700ead-89a9-4ee7-a12a-934b6c5cb2cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused"
Nov 01 05:51:33.339668 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:33.339635    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:51:33.594549 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:33.593957    5689 patch_prober.go:29] interesting pod/e2e-mt-25565930-openshift-default-64f5666649-cpzlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body=
Nov 01 05:51:33.594549 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:33.594042    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-cpzlj" podUID="75700ead-89a9-4ee7-a12a-934b6c5cb2cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused"
Nov 01 05:51:34.597408 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:34.597374    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-cpzlj"
Nov 01 05:51:34.597872 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:34.597683    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-cpzlj"
Nov 01 05:51:34.598421 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:34.598401    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-cpzlj"
Nov 01 05:51:34.609854 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:34.609816    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:39.316190 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:39.316156    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:44.046868 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:44.046826    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:48.760836 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:48.760799    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:53.528043 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:53.527898    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:58.440067 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:58.440034    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:59.649317 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:59.649282    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx"]
Nov 01 05:51:59.654725 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:59.654705    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx"
Nov 01 05:51:59.760888 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:59.760856    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx"]
Nov 01 05:51:59.782000 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:59.781966    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cc52d9e-b04b-479b-b128-ece05b493ae9-istio-envoy\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx\" (UID: \"9cc52d9e-b04b-479b-b128-ece05b493ae9\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx"
Nov 01 05:51:59.782000 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:59.782005    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzgcq\" (UniqueName: \"kubernetes.io/projected/9cc52d9e-b04b-479b-b128-ece05b493ae9-kube-api-access-jzgcq\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx\" (UID: \"9cc52d9e-b04b-479b-b128-ece05b493ae9\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx"
Nov 01 05:51:59.782205 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:59.782037    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cc52d9e-b04b-479b-b128-ece05b493ae9-workload-socket\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx\" (UID: \"9cc52d9e-b04b-479b-b128-ece05b493ae9\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx"
Nov 01 05:51:59.782205 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:59.782179    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cc52d9e-b04b-479b-b128-ece05b493ae9-workload-certs\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx\" (UID: \"9cc52d9e-b04b-479b-b128-ece05b493ae9\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx"
Nov 01 05:51:59.782296 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:59.782283    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cc52d9e-b04b-479b-b128-ece05b493ae9-credential-socket\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx\" (UID: \"9cc52d9e-b04b-479b-b128-ece05b493ae9\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx"
Nov 01 05:51:59.782408 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:59.782395    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cc52d9e-b04b-479b-b128-ece05b493ae9-istio-token\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx\" (UID: \"9cc52d9e-b04b-479b-b128-ece05b493ae9\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx"
Nov 01 05:51:59.782479 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:59.782427    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cc52d9e-b04b-479b-b128-ece05b493ae9-istio-data\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx\" (UID: \"9cc52d9e-b04b-479b-b128-ece05b493ae9\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx"
Nov 01 05:51:59.782533 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:59.782499    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cc52d9e-b04b-479b-b128-ece05b493ae9-istio-podinfo\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx\" (UID: \"9cc52d9e-b04b-479b-b128-ece05b493ae9\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx"
Nov 01 05:51:59.782577 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:59.782552    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cc52d9e-b04b-479b-b128-ece05b493ae9-istiod-ca-cert\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx\" (UID: \"9cc52d9e-b04b-479b-b128-ece05b493ae9\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx"
Nov 01 05:51:59.884078 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:59.884048    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cc52d9e-b04b-479b-b128-ece05b493ae9-workload-certs\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx\" (UID: \"9cc52d9e-b04b-479b-b128-ece05b493ae9\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx"
Nov 01 05:51:59.884243 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:59.884103    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cc52d9e-b04b-479b-b128-ece05b493ae9-credential-socket\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx\" (UID: \"9cc52d9e-b04b-479b-b128-ece05b493ae9\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx"
Nov 01 05:51:59.884243 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:59.884161    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cc52d9e-b04b-479b-b128-ece05b493ae9-istio-token\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx\" (UID: \"9cc52d9e-b04b-479b-b128-ece05b493ae9\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx"
Nov 01 05:51:59.884243 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:59.884187    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cc52d9e-b04b-479b-b128-ece05b493ae9-istio-data\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx\" (UID: \"9cc52d9e-b04b-479b-b128-ece05b493ae9\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx"
Nov 01 05:51:59.884404 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:59.884311    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cc52d9e-b04b-479b-b128-ece05b493ae9-istio-podinfo\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx\" (UID: \"9cc52d9e-b04b-479b-b128-ece05b493ae9\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx"
Nov 01 05:51:59.884404 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:59.884347    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cc52d9e-b04b-479b-b128-ece05b493ae9-istiod-ca-cert\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx\" (UID: \"9cc52d9e-b04b-479b-b128-ece05b493ae9\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx"
Nov 01 05:51:59.884404 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:59.884399    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cc52d9e-b04b-479b-b128-ece05b493ae9-istio-envoy\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx\" (UID: \"9cc52d9e-b04b-479b-b128-ece05b493ae9\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx"
Nov 01 05:51:59.884609 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:59.884429    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzgcq\" (UniqueName: \"kubernetes.io/projected/9cc52d9e-b04b-479b-b128-ece05b493ae9-kube-api-access-jzgcq\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx\" (UID: \"9cc52d9e-b04b-479b-b128-ece05b493ae9\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx"
Nov 01 05:51:59.884609 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:59.884500    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cc52d9e-b04b-479b-b128-ece05b493ae9-workload-certs\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx\" (UID: \"9cc52d9e-b04b-479b-b128-ece05b493ae9\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx"
Nov 01 05:51:59.884609 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:59.884504    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cc52d9e-b04b-479b-b128-ece05b493ae9-workload-socket\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx\" (UID: \"9cc52d9e-b04b-479b-b128-ece05b493ae9\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx"
Nov 01 05:51:59.884609 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:59.884525    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cc52d9e-b04b-479b-b128-ece05b493ae9-credential-socket\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx\" (UID: \"9cc52d9e-b04b-479b-b128-ece05b493ae9\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx"
Nov 01 05:51:59.884609 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:59.884557    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cc52d9e-b04b-479b-b128-ece05b493ae9-istio-data\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx\" (UID: \"9cc52d9e-b04b-479b-b128-ece05b493ae9\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx"
Nov 01 05:51:59.884878 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:59.884855    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cc52d9e-b04b-479b-b128-ece05b493ae9-workload-socket\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx\" (UID: \"9cc52d9e-b04b-479b-b128-ece05b493ae9\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx"
Nov 01 05:51:59.885150 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:59.885126    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cc52d9e-b04b-479b-b128-ece05b493ae9-istiod-ca-cert\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx\" (UID: \"9cc52d9e-b04b-479b-b128-ece05b493ae9\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx"
Nov 01 05:51:59.887106 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:59.887055    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cc52d9e-b04b-479b-b128-ece05b493ae9-istio-envoy\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx\" (UID: \"9cc52d9e-b04b-479b-b128-ece05b493ae9\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx"
Nov 01 05:51:59.887386 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:59.887366    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cc52d9e-b04b-479b-b128-ece05b493ae9-istio-podinfo\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx\" (UID: \"9cc52d9e-b04b-479b-b128-ece05b493ae9\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx"
Nov 01 05:51:59.944920 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:59.944848    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cc52d9e-b04b-479b-b128-ece05b493ae9-istio-token\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx\" (UID: \"9cc52d9e-b04b-479b-b128-ece05b493ae9\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx"
Nov 01 05:51:59.944920 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:59.944898    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzgcq\" (UniqueName: \"kubernetes.io/projected/9cc52d9e-b04b-479b-b128-ece05b493ae9-kube-api-access-jzgcq\") pod \"e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx\" (UID: \"9cc52d9e-b04b-479b-b128-ece05b493ae9\") " pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx"
Nov 01 05:51:59.968154 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:51:59.968133    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx"
Nov 01 05:52:00.074410 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:52:00.074371    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:52:00.400484 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:52:00.400463    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx"]
Nov 01 05:52:00.403194 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:52:00.403168    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cc52d9e_b04b_479b_b128_ece05b493ae9.slice/crio-65a074e1c635a58dcc2fe4b03280d6d93214b7b6adcbd0ab23de301c3cd31c83 WatchSource:0}: Error finding container 65a074e1c635a58dcc2fe4b03280d6d93214b7b6adcbd0ab23de301c3cd31c83: Status 404 returned error can't find the container with id 65a074e1c635a58dcc2fe4b03280d6d93214b7b6adcbd0ab23de301c3cd31c83
Nov 01 05:52:00.405510 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:52:00.405480    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:52:00.405605 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:52:00.405557    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:52:00.405663 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:52:00.405602    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 05:52:00.562585 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:52:00.562545    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:52:01.025298 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:52:01.025263    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx" event={"ID":"9cc52d9e-b04b-479b-b128-ece05b493ae9","Type":"ContainerStarted","Data":"75d2cc387aff6a8cfc278d5182c9ba395983507f59b2d55722e4d4bdabfb26b0"}
Nov 01 05:52:01.025719 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:52:01.025313    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx" event={"ID":"9cc52d9e-b04b-479b-b128-ece05b493ae9","Type":"ContainerStarted","Data":"65a074e1c635a58dcc2fe4b03280d6d93214b7b6adcbd0ab23de301c3cd31c83"}
Nov 01 05:52:01.160388 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:52:01.160324    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx" podStartSLOduration=2.160305353 podStartE2EDuration="2.160305353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:51:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:52:01.156880649 +0000 UTC m=+2899.227677368" watchObservedRunningTime="2026-11-01 05:52:01.160305353 +0000 UTC m=+2899.231102073"
Nov 01 05:52:01.968864 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:52:01.968832    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx"
Nov 01 05:52:01.970515 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:52:01.970485    5689 patch_prober.go:29] interesting pod/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Nov 01 05:52:01.970624 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:52:01.970544    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx" podUID="9cc52d9e-b04b-479b-b128-ece05b493ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Nov 01 05:52:02.255443 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:52:02.255362    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:52:02.986479 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:52:02.969948    5689 patch_prober.go:29] interesting pod/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Nov 01 05:52:02.986479 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:52:02.970009    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx" podUID="9cc52d9e-b04b-479b-b128-ece05b493ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Nov 01 05:52:03.800711 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:52:03.800685    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:52:03.978480 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:52:03.970874    5689 patch_prober.go:29] interesting pod/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Nov 01 05:52:03.978480 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:52:03.970940    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx" podUID="9cc52d9e-b04b-479b-b128-ece05b493ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Nov 01 05:52:04.086555 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:52:04.086406    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:52:04.969210 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:52:04.969178    5689 patch_prober.go:29] interesting pod/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Nov 01 05:52:04.969717 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:52:04.969235    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx" podUID="9cc52d9e-b04b-479b-b128-ece05b493ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Nov 01 05:52:05.969301 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:52:05.969268    5689 patch_prober.go:29] interesting pod/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Nov 01 05:52:05.969715 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:52:05.969319    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx" podUID="9cc52d9e-b04b-479b-b128-ece05b493ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Nov 01 05:52:06.974749 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:52:06.974718    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx"
Nov 01 05:52:06.975204 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:52:06.975175    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx"
Nov 01 05:52:06.975928 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:52:06.975909    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx"
Nov 01 05:52:08.883370 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:52:08.883339    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:52:09.098855 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:52:09.098821    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:52:10.172025 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:52:10.171997    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:52:10.346511 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:52:10.346479    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:52:13.363321 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:52:13.363287    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:52:13.724966 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:52:13.724933    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:52:13.934214 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:52:13.934169    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:52:14.111085 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:52:14.110997    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:52:14.381691 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:52:14.381618    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:52:18.601887 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:52:18.601857    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:52:19.437566 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:52:19.437531    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:52:23.428483 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:52:23.423221    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:52:28.157842 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:52:28.151086    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:52:30.070673 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:52:30.070641    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:52:32.898805 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:52:32.898777    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:52:37.716623 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:52:37.716589    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:52:42.495600 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:52:42.495568    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:52:43.232392 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:52:43.232359    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:52:43.551546 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:52:43.551471    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:52:47.199037 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:52:47.199005    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:52:52.038993 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:52:52.038955    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:52:53.159769 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:52:53.159731    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:52:55.799885 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:52:55.799855    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-25565930-5c94467bfc-hp4rt"]
Nov 01 05:52:55.907647 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:52:55.907609    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-25565930-b8577d659-6pwlj"]
Nov 01 05:52:56.201206 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:52:56.201176    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:52:56.406478 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:52:56.398627    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:52:56.591667 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:52:56.582387    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:52:56.858265 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:52:56.858188    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:52:57.029850 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:52:57.029813    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:53:00.068850 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:53:00.068814    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:53:00.800860 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:00.800815    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-25565930-5c94467bfc-hp4rt" podUID="0e6b057a-6ebd-4a97-8ef9-e2c77f0379c0" containerName="payload-processing" containerID="cri-o://1b75994dd0b73c7f3d7021be0a0c7bd45a9a0763a4f283c005ee0a020a712a35" gracePeriod=25
Nov 01 05:53:00.908618 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:00.908564    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-25565930-b8577d659-6pwlj" podUID="5b679696-4dae-41d8-bc1f-1a598aeedec9" containerName="payload-pre-processing" containerID="cri-o://ec727af83bc419dc7e4c91e383b23efef072d4a143e5141fa6176d7c46f69120" gracePeriod=25
Nov 01 05:53:01.093247 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:01.093219    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-25565930-5c94467bfc-hp4rt"
Nov 01 05:53:01.141183 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:01.141149    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e6b057a-6ebd-4a97-8ef9-e2c77f0379c0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e6b057a-6ebd-4a97-8ef9-e2c77f0379c0-plugins-config-volume\") pod \"0e6b057a-6ebd-4a97-8ef9-e2c77f0379c0\" (UID: \"0e6b057a-6ebd-4a97-8ef9-e2c77f0379c0\") "
Nov 01 05:53:01.141410 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:01.141386    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e6b057a-6ebd-4a97-8ef9-e2c77f0379c0-kube-api-access-mrhqh\" (UniqueName: \"kubernetes.io/projected/0e6b057a-6ebd-4a97-8ef9-e2c77f0379c0-kube-api-access-mrhqh\") pod \"0e6b057a-6ebd-4a97-8ef9-e2c77f0379c0\" (UID: \"0e6b057a-6ebd-4a97-8ef9-e2c77f0379c0\") "
Nov 01 05:53:01.141593 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:01.141567    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e6b057a-6ebd-4a97-8ef9-e2c77f0379c0-plugins-config-volume" pod "0e6b057a-6ebd-4a97-8ef9-e2c77f0379c0" (UID: "0e6b057a-6ebd-4a97-8ef9-e2c77f0379c0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:53:01.142053 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:01.142027    5689 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e6b057a-6ebd-4a97-8ef9-e2c77f0379c0-plugins-config-volume\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:53:01.143697 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:01.143670    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e6b057a-6ebd-4a97-8ef9-e2c77f0379c0-kube-api-access-mrhqh" pod "0e6b057a-6ebd-4a97-8ef9-e2c77f0379c0" (UID: "0e6b057a-6ebd-4a97-8ef9-e2c77f0379c0"). InnerVolumeSpecName "kube-api-access-mrhqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:53:01.182328 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:01.182308    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-25565930-b8577d659-6pwlj"
Nov 01 05:53:01.242837 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:01.242811    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b679696-4dae-41d8-bc1f-1a598aeedec9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b679696-4dae-41d8-bc1f-1a598aeedec9-plugins-config-volume\") pod \"5b679696-4dae-41d8-bc1f-1a598aeedec9\" (UID: \"5b679696-4dae-41d8-bc1f-1a598aeedec9\") "
Nov 01 05:53:01.242978 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:01.242874    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b679696-4dae-41d8-bc1f-1a598aeedec9-kube-api-access-6g722\" (UniqueName: \"kubernetes.io/projected/5b679696-4dae-41d8-bc1f-1a598aeedec9-kube-api-access-6g722\") pod \"5b679696-4dae-41d8-bc1f-1a598aeedec9\" (UID: \"5b679696-4dae-41d8-bc1f-1a598aeedec9\") "
Nov 01 05:53:01.243241 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:01.243218    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b679696-4dae-41d8-bc1f-1a598aeedec9-plugins-config-volume" pod "5b679696-4dae-41d8-bc1f-1a598aeedec9" (UID: "5b679696-4dae-41d8-bc1f-1a598aeedec9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:53:01.243520 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:01.243502    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrhqh\" (UniqueName: \"kubernetes.io/projected/0e6b057a-6ebd-4a97-8ef9-e2c77f0379c0-kube-api-access-mrhqh\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:53:01.243768 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:01.243521    5689 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b679696-4dae-41d8-bc1f-1a598aeedec9-plugins-config-volume\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:53:01.245124 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:01.245094    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b679696-4dae-41d8-bc1f-1a598aeedec9-kube-api-access-6g722" pod "5b679696-4dae-41d8-bc1f-1a598aeedec9" (UID: "5b679696-4dae-41d8-bc1f-1a598aeedec9"). InnerVolumeSpecName "kube-api-access-6g722". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:53:01.345701 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:01.345631    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6g722\" (UniqueName: \"kubernetes.io/projected/5b679696-4dae-41d8-bc1f-1a598aeedec9-kube-api-access-6g722\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:53:01.490573 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:01.490532    5689 generic.go:356] "Generic (PLEG): container finished" podID="0e6b057a-6ebd-4a97-8ef9-e2c77f0379c0" containerID="1b75994dd0b73c7f3d7021be0a0c7bd45a9a0763a4f283c005ee0a020a712a35" exitCode=0
Nov 01 05:53:01.490573 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:01.490569    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-25565930-5c94467bfc-hp4rt" event={"ID":"0e6b057a-6ebd-4a97-8ef9-e2c77f0379c0","Type":"ContainerDied","Data":"1b75994dd0b73c7f3d7021be0a0c7bd45a9a0763a4f283c005ee0a020a712a35"}
Nov 01 05:53:01.490886 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:01.490630    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-25565930-5c94467bfc-hp4rt" event={"ID":"0e6b057a-6ebd-4a97-8ef9-e2c77f0379c0","Type":"ContainerDied","Data":"15665e7b1da0fb856ddaee028d6136b45b157d21dd82ffc0fedd85260ad2a889"}
Nov 01 05:53:01.490886 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:01.490583    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-25565930-5c94467bfc-hp4rt"
Nov 01 05:53:01.490886 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:01.490657    5689 scope.go:122] "RemoveContainer" containerID="1b75994dd0b73c7f3d7021be0a0c7bd45a9a0763a4f283c005ee0a020a712a35"
Nov 01 05:53:01.492343 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:01.492320    5689 generic.go:356] "Generic (PLEG): container finished" podID="5b679696-4dae-41d8-bc1f-1a598aeedec9" containerID="ec727af83bc419dc7e4c91e383b23efef072d4a143e5141fa6176d7c46f69120" exitCode=0
Nov 01 05:53:01.492511 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:01.492494    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-25565930-b8577d659-6pwlj" event={"ID":"5b679696-4dae-41d8-bc1f-1a598aeedec9","Type":"ContainerDied","Data":"ec727af83bc419dc7e4c91e383b23efef072d4a143e5141fa6176d7c46f69120"}
Nov 01 05:53:01.492588 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:01.492538    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-25565930-b8577d659-6pwlj" event={"ID":"5b679696-4dae-41d8-bc1f-1a598aeedec9","Type":"ContainerDied","Data":"f31f54621fc77069ede8dd2ed5f45076b76871f5b56ae67a16db19f2525f8956"}
Nov 01 05:53:01.492654 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:01.492608    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-25565930-b8577d659-6pwlj"
Nov 01 05:53:01.501429 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:01.501413    5689 scope.go:122] "RemoveContainer" containerID="1b75994dd0b73c7f3d7021be0a0c7bd45a9a0763a4f283c005ee0a020a712a35"
Nov 01 05:53:01.502219 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:53:01.502190    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b75994dd0b73c7f3d7021be0a0c7bd45a9a0763a4f283c005ee0a020a712a35\": container with ID starting with 1b75994dd0b73c7f3d7021be0a0c7bd45a9a0763a4f283c005ee0a020a712a35 not found: ID does not exist" containerID="1b75994dd0b73c7f3d7021be0a0c7bd45a9a0763a4f283c005ee0a020a712a35"
Nov 01 05:53:01.502306 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:01.502227    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b75994dd0b73c7f3d7021be0a0c7bd45a9a0763a4f283c005ee0a020a712a35"} err="failed to get container status \"1b75994dd0b73c7f3d7021be0a0c7bd45a9a0763a4f283c005ee0a020a712a35\": rpc error: code = NotFound desc = could not find container \"1b75994dd0b73c7f3d7021be0a0c7bd45a9a0763a4f283c005ee0a020a712a35\": container with ID starting with 1b75994dd0b73c7f3d7021be0a0c7bd45a9a0763a4f283c005ee0a020a712a35 not found: ID does not exist"
Nov 01 05:53:01.502306 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:01.502243    5689 scope.go:122] "RemoveContainer" containerID="ec727af83bc419dc7e4c91e383b23efef072d4a143e5141fa6176d7c46f69120"
Nov 01 05:53:01.511351 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:01.511337    5689 scope.go:122] "RemoveContainer" containerID="ec727af83bc419dc7e4c91e383b23efef072d4a143e5141fa6176d7c46f69120"
Nov 01 05:53:01.511641 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:53:01.511611    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec727af83bc419dc7e4c91e383b23efef072d4a143e5141fa6176d7c46f69120\": container with ID starting with ec727af83bc419dc7e4c91e383b23efef072d4a143e5141fa6176d7c46f69120 not found: ID does not exist" containerID="ec727af83bc419dc7e4c91e383b23efef072d4a143e5141fa6176d7c46f69120"
Nov 01 05:53:01.511694 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:01.511648    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec727af83bc419dc7e4c91e383b23efef072d4a143e5141fa6176d7c46f69120"} err="failed to get container status \"ec727af83bc419dc7e4c91e383b23efef072d4a143e5141fa6176d7c46f69120\": rpc error: code = NotFound desc = could not find container \"ec727af83bc419dc7e4c91e383b23efef072d4a143e5141fa6176d7c46f69120\": container with ID starting with ec727af83bc419dc7e4c91e383b23efef072d4a143e5141fa6176d7c46f69120 not found: ID does not exist"
Nov 01 05:53:01.655360 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:01.655331    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-25565930-5c94467bfc-hp4rt"]
Nov 01 05:53:01.658880 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:01.658857    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-25565930-5c94467bfc-hp4rt"]
Nov 01 05:53:01.766791 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:01.766759    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-25565930-b8577d659-6pwlj"]
Nov 01 05:53:01.770706 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:01.770681    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-25565930-b8577d659-6pwlj"]
Nov 01 05:53:01.991886 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:01.991819    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:53:02.073486 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:02.073414    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e6b057a-6ebd-4a97-8ef9-e2c77f0379c0" path="/var/lib/kubelet/pods/0e6b057a-6ebd-4a97-8ef9-e2c77f0379c0/volumes"
Nov 01 05:53:02.074057 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:02.074036    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b679696-4dae-41d8-bc1f-1a598aeedec9" path="/var/lib/kubelet/pods/5b679696-4dae-41d8-bc1f-1a598aeedec9/volumes"
Nov 01 05:53:02.686941 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:02.686910    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:53:04.062096 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:04.062065    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:53:04.233018 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:04.232977    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:53:06.183748 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:06.183719    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:53:08.016407 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:08.016380    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:53:08.435940 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:08.435900    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:53:08.613660 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:08.613625    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:53:10.016153 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:10.016123    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:53:10.596352 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:10.596316    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:53:10.604410 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:10.604385    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-cpzlj"]
Nov 01 05:53:10.604622 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:10.604599    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-cpzlj" podUID="75700ead-89a9-4ee7-a12a-934b6c5cb2cb" containerName="istio-proxy" containerID="cri-o://535e7da13a60fc5686a3870e965895709a6a154b689ef823b37d7ebadca43938" gracePeriod=30
Nov 01 05:53:10.615869 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:10.615843    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gbhn9"]
Nov 01 05:53:10.616058 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:10.616038    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gbhn9" podUID="e67e1922-ea2b-4fbc-858b-1648ec6124e9" containerName="istio-proxy" containerID="cri-o://7ae3b46ccabc42915dfdc51dafc01429fa1d2b7fb7e917fe70904e176d03fb08" gracePeriod=30
Nov 01 05:53:10.806498 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:10.802536    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:53:13.518512 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:13.515260    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:53:15.879317 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:15.879248    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-cpzlj"
Nov 01 05:53:15.927996 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:15.927972    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gbhn9"
Nov 01 05:53:15.941630 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:15.941605    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75700ead-89a9-4ee7-a12a-934b6c5cb2cb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75700ead-89a9-4ee7-a12a-934b6c5cb2cb-credential-socket\") pod \"75700ead-89a9-4ee7-a12a-934b6c5cb2cb\" (UID: \"75700ead-89a9-4ee7-a12a-934b6c5cb2cb\") "
Nov 01 05:53:15.941731 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:15.941649    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/75700ead-89a9-4ee7-a12a-934b6c5cb2cb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75700ead-89a9-4ee7-a12a-934b6c5cb2cb-istio-podinfo\") pod \"75700ead-89a9-4ee7-a12a-934b6c5cb2cb\" (UID: \"75700ead-89a9-4ee7-a12a-934b6c5cb2cb\") "
Nov 01 05:53:15.941731 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:15.941682    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75700ead-89a9-4ee7-a12a-934b6c5cb2cb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75700ead-89a9-4ee7-a12a-934b6c5cb2cb-workload-socket\") pod \"75700ead-89a9-4ee7-a12a-934b6c5cb2cb\" (UID: \"75700ead-89a9-4ee7-a12a-934b6c5cb2cb\") "
Nov 01 05:53:15.941851 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:15.941759    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75700ead-89a9-4ee7-a12a-934b6c5cb2cb-istio-token\" (UniqueName: \"kubernetes.io/projected/75700ead-89a9-4ee7-a12a-934b6c5cb2cb-istio-token\") pod \"75700ead-89a9-4ee7-a12a-934b6c5cb2cb\" (UID: \"75700ead-89a9-4ee7-a12a-934b6c5cb2cb\") "
Nov 01 05:53:15.941851 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:15.941825    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75700ead-89a9-4ee7-a12a-934b6c5cb2cb-credential-socket" pod "75700ead-89a9-4ee7-a12a-934b6c5cb2cb" (UID: "75700ead-89a9-4ee7-a12a-934b6c5cb2cb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:53:15.941966 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:15.941849    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75700ead-89a9-4ee7-a12a-934b6c5cb2cb-kube-api-access-cdshb\" (UniqueName: \"kubernetes.io/projected/75700ead-89a9-4ee7-a12a-934b6c5cb2cb-kube-api-access-cdshb\") pod \"75700ead-89a9-4ee7-a12a-934b6c5cb2cb\" (UID: \"75700ead-89a9-4ee7-a12a-934b6c5cb2cb\") "
Nov 01 05:53:15.941966 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:15.941894    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75700ead-89a9-4ee7-a12a-934b6c5cb2cb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75700ead-89a9-4ee7-a12a-934b6c5cb2cb-istio-envoy\") pod \"75700ead-89a9-4ee7-a12a-934b6c5cb2cb\" (UID: \"75700ead-89a9-4ee7-a12a-934b6c5cb2cb\") "
Nov 01 05:53:15.941966 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:15.941925    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75700ead-89a9-4ee7-a12a-934b6c5cb2cb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75700ead-89a9-4ee7-a12a-934b6c5cb2cb-workload-certs\") pod \"75700ead-89a9-4ee7-a12a-934b6c5cb2cb\" (UID: \"75700ead-89a9-4ee7-a12a-934b6c5cb2cb\") "
Nov 01 05:53:15.942114 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:15.941975    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75700ead-89a9-4ee7-a12a-934b6c5cb2cb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75700ead-89a9-4ee7-a12a-934b6c5cb2cb-istio-data\") pod \"75700ead-89a9-4ee7-a12a-934b6c5cb2cb\" (UID: \"75700ead-89a9-4ee7-a12a-934b6c5cb2cb\") "
Nov 01 05:53:15.942114 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:15.942052    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75700ead-89a9-4ee7-a12a-934b6c5cb2cb-workload-socket" pod "75700ead-89a9-4ee7-a12a-934b6c5cb2cb" (UID: "75700ead-89a9-4ee7-a12a-934b6c5cb2cb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:53:15.942114 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:15.942087    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75700ead-89a9-4ee7-a12a-934b6c5cb2cb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75700ead-89a9-4ee7-a12a-934b6c5cb2cb-istiod-ca-cert\") pod \"75700ead-89a9-4ee7-a12a-934b6c5cb2cb\" (UID: \"75700ead-89a9-4ee7-a12a-934b6c5cb2cb\") "
Nov 01 05:53:15.942285 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:15.942264    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75700ead-89a9-4ee7-a12a-934b6c5cb2cb-istio-data" pod "75700ead-89a9-4ee7-a12a-934b6c5cb2cb" (UID: "75700ead-89a9-4ee7-a12a-934b6c5cb2cb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:53:15.942715 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:15.942690    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75700ead-89a9-4ee7-a12a-934b6c5cb2cb-istiod-ca-cert" pod "75700ead-89a9-4ee7-a12a-934b6c5cb2cb" (UID: "75700ead-89a9-4ee7-a12a-934b6c5cb2cb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:53:15.942960 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:15.942942    5689 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75700ead-89a9-4ee7-a12a-934b6c5cb2cb-workload-socket\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:53:15.943075 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:15.942969    5689 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75700ead-89a9-4ee7-a12a-934b6c5cb2cb-istio-data\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:53:15.943075 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:15.942985    5689 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75700ead-89a9-4ee7-a12a-934b6c5cb2cb-istiod-ca-cert\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:53:15.943075 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:15.942999    5689 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75700ead-89a9-4ee7-a12a-934b6c5cb2cb-credential-socket\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:53:15.943075 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:15.942204    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75700ead-89a9-4ee7-a12a-934b6c5cb2cb-workload-certs" pod "75700ead-89a9-4ee7-a12a-934b6c5cb2cb" (UID: "75700ead-89a9-4ee7-a12a-934b6c5cb2cb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:53:15.944047 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:15.944026    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75700ead-89a9-4ee7-a12a-934b6c5cb2cb-kube-api-access-cdshb" pod "75700ead-89a9-4ee7-a12a-934b6c5cb2cb" (UID: "75700ead-89a9-4ee7-a12a-934b6c5cb2cb"). InnerVolumeSpecName "kube-api-access-cdshb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:53:15.944118 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:15.944083    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75700ead-89a9-4ee7-a12a-934b6c5cb2cb-istio-token" pod "75700ead-89a9-4ee7-a12a-934b6c5cb2cb" (UID: "75700ead-89a9-4ee7-a12a-934b6c5cb2cb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:53:15.944491 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:15.944438    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75700ead-89a9-4ee7-a12a-934b6c5cb2cb-istio-envoy" pod "75700ead-89a9-4ee7-a12a-934b6c5cb2cb" (UID: "75700ead-89a9-4ee7-a12a-934b6c5cb2cb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:53:15.944563 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:15.944436    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/75700ead-89a9-4ee7-a12a-934b6c5cb2cb-istio-podinfo" pod "75700ead-89a9-4ee7-a12a-934b6c5cb2cb" (UID: "75700ead-89a9-4ee7-a12a-934b6c5cb2cb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 05:53:16.043408 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:16.043384    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e67e1922-ea2b-4fbc-858b-1648ec6124e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e67e1922-ea2b-4fbc-858b-1648ec6124e9-istio-data\") pod \"e67e1922-ea2b-4fbc-858b-1648ec6124e9\" (UID: \"e67e1922-ea2b-4fbc-858b-1648ec6124e9\") "
Nov 01 05:53:16.043544 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:16.043423    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e67e1922-ea2b-4fbc-858b-1648ec6124e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e67e1922-ea2b-4fbc-858b-1648ec6124e9-istiod-ca-cert\") pod \"e67e1922-ea2b-4fbc-858b-1648ec6124e9\" (UID: \"e67e1922-ea2b-4fbc-858b-1648ec6124e9\") "
Nov 01 05:53:16.043544 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:16.043443    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e67e1922-ea2b-4fbc-858b-1648ec6124e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e67e1922-ea2b-4fbc-858b-1648ec6124e9-workload-certs\") pod \"e67e1922-ea2b-4fbc-858b-1648ec6124e9\" (UID: \"e67e1922-ea2b-4fbc-858b-1648ec6124e9\") "
Nov 01 05:53:16.043544 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:16.043506    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e67e1922-ea2b-4fbc-858b-1648ec6124e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e67e1922-ea2b-4fbc-858b-1648ec6124e9-workload-socket\") pod \"e67e1922-ea2b-4fbc-858b-1648ec6124e9\" (UID: \"e67e1922-ea2b-4fbc-858b-1648ec6124e9\") "
Nov 01 05:53:16.043683 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:16.043578    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e67e1922-ea2b-4fbc-858b-1648ec6124e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e67e1922-ea2b-4fbc-858b-1648ec6124e9-credential-socket\") pod \"e67e1922-ea2b-4fbc-858b-1648ec6124e9\" (UID: \"e67e1922-ea2b-4fbc-858b-1648ec6124e9\") "
Nov 01 05:53:16.043683 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:16.043641    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e67e1922-ea2b-4fbc-858b-1648ec6124e9-istio-data" pod "e67e1922-ea2b-4fbc-858b-1648ec6124e9" (UID: "e67e1922-ea2b-4fbc-858b-1648ec6124e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:53:16.043683 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:16.043658    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e67e1922-ea2b-4fbc-858b-1648ec6124e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e67e1922-ea2b-4fbc-858b-1648ec6124e9-istio-podinfo\") pod \"e67e1922-ea2b-4fbc-858b-1648ec6124e9\" (UID: \"e67e1922-ea2b-4fbc-858b-1648ec6124e9\") "
Nov 01 05:53:16.043826 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:16.043726    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e67e1922-ea2b-4fbc-858b-1648ec6124e9-kube-api-access-5pv82\" (UniqueName: \"kubernetes.io/projected/e67e1922-ea2b-4fbc-858b-1648ec6124e9-kube-api-access-5pv82\") pod \"e67e1922-ea2b-4fbc-858b-1648ec6124e9\" (UID: \"e67e1922-ea2b-4fbc-858b-1648ec6124e9\") "
Nov 01 05:53:16.043826 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:16.043798    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e67e1922-ea2b-4fbc-858b-1648ec6124e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e67e1922-ea2b-4fbc-858b-1648ec6124e9-istio-envoy\") pod \"e67e1922-ea2b-4fbc-858b-1648ec6124e9\" (UID: \"e67e1922-ea2b-4fbc-858b-1648ec6124e9\") "
Nov 01 05:53:16.043932 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:16.043828    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e67e1922-ea2b-4fbc-858b-1648ec6124e9-credential-socket" pod "e67e1922-ea2b-4fbc-858b-1648ec6124e9" (UID: "e67e1922-ea2b-4fbc-858b-1648ec6124e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:53:16.043932 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:16.043841    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e67e1922-ea2b-4fbc-858b-1648ec6124e9-workload-socket" pod "e67e1922-ea2b-4fbc-858b-1648ec6124e9" (UID: "e67e1922-ea2b-4fbc-858b-1648ec6124e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:53:16.043932 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:16.043854    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e67e1922-ea2b-4fbc-858b-1648ec6124e9-workload-certs" pod "e67e1922-ea2b-4fbc-858b-1648ec6124e9" (UID: "e67e1922-ea2b-4fbc-858b-1648ec6124e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:53:16.043932 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:16.043863    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e67e1922-ea2b-4fbc-858b-1648ec6124e9-istiod-ca-cert" pod "e67e1922-ea2b-4fbc-858b-1648ec6124e9" (UID: "e67e1922-ea2b-4fbc-858b-1648ec6124e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:53:16.043932 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:16.043877    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e67e1922-ea2b-4fbc-858b-1648ec6124e9-istio-token\" (UniqueName: \"kubernetes.io/projected/e67e1922-ea2b-4fbc-858b-1648ec6124e9-istio-token\") pod \"e67e1922-ea2b-4fbc-858b-1648ec6124e9\" (UID: \"e67e1922-ea2b-4fbc-858b-1648ec6124e9\") "
Nov 01 05:53:16.044622 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:16.044524    5689 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e67e1922-ea2b-4fbc-858b-1648ec6124e9-istio-data\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:53:16.044736 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:16.044625    5689 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e67e1922-ea2b-4fbc-858b-1648ec6124e9-istiod-ca-cert\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:53:16.044736 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:16.044642    5689 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e67e1922-ea2b-4fbc-858b-1648ec6124e9-workload-certs\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:53:16.044736 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:16.044658    5689 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e67e1922-ea2b-4fbc-858b-1648ec6124e9-workload-socket\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:53:16.044736 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:16.044674    5689 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75700ead-89a9-4ee7-a12a-934b6c5cb2cb-istio-podinfo\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:53:16.044736 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:16.044688    5689 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e67e1922-ea2b-4fbc-858b-1648ec6124e9-credential-socket\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:53:16.044736 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:16.044702    5689 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75700ead-89a9-4ee7-a12a-934b6c5cb2cb-istio-token\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:53:16.044736 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:16.044717    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdshb\" (UniqueName: \"kubernetes.io/projected/75700ead-89a9-4ee7-a12a-934b6c5cb2cb-kube-api-access-cdshb\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:53:16.044736 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:16.044732    5689 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75700ead-89a9-4ee7-a12a-934b6c5cb2cb-istio-envoy\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:53:16.045169 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:16.044747    5689 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75700ead-89a9-4ee7-a12a-934b6c5cb2cb-workload-certs\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:53:16.045822 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:16.045801    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e67e1922-ea2b-4fbc-858b-1648ec6124e9-istio-podinfo" pod "e67e1922-ea2b-4fbc-858b-1648ec6124e9" (UID: "e67e1922-ea2b-4fbc-858b-1648ec6124e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 05:53:16.045940 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:16.045921    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e67e1922-ea2b-4fbc-858b-1648ec6124e9-kube-api-access-5pv82" pod "e67e1922-ea2b-4fbc-858b-1648ec6124e9" (UID: "e67e1922-ea2b-4fbc-858b-1648ec6124e9"). InnerVolumeSpecName "kube-api-access-5pv82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:53:16.046006 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:16.045934    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e67e1922-ea2b-4fbc-858b-1648ec6124e9-istio-token" pod "e67e1922-ea2b-4fbc-858b-1648ec6124e9" (UID: "e67e1922-ea2b-4fbc-858b-1648ec6124e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:53:16.046077 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:16.046063    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e67e1922-ea2b-4fbc-858b-1648ec6124e9-istio-envoy" pod "e67e1922-ea2b-4fbc-858b-1648ec6124e9" (UID: "e67e1922-ea2b-4fbc-858b-1648ec6124e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:53:16.146152 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:16.146075    5689 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e67e1922-ea2b-4fbc-858b-1648ec6124e9-istio-token\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:53:16.146152 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:16.146100    5689 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e67e1922-ea2b-4fbc-858b-1648ec6124e9-istio-podinfo\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:53:16.146152 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:16.146110    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pv82\" (UniqueName: \"kubernetes.io/projected/e67e1922-ea2b-4fbc-858b-1648ec6124e9-kube-api-access-5pv82\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:53:16.146152 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:16.146121    5689 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e67e1922-ea2b-4fbc-858b-1648ec6124e9-istio-envoy\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:53:16.693389 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:16.693351    5689 generic.go:356] "Generic (PLEG): container finished" podID="e67e1922-ea2b-4fbc-858b-1648ec6124e9" containerID="7ae3b46ccabc42915dfdc51dafc01429fa1d2b7fb7e917fe70904e176d03fb08" exitCode=0
Nov 01 05:53:16.693609 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:16.693400    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gbhn9"
Nov 01 05:53:16.693609 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:16.693444    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gbhn9" event={"ID":"e67e1922-ea2b-4fbc-858b-1648ec6124e9","Type":"ContainerDied","Data":"7ae3b46ccabc42915dfdc51dafc01429fa1d2b7fb7e917fe70904e176d03fb08"}
Nov 01 05:53:16.693609 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:16.693521    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gbhn9" event={"ID":"e67e1922-ea2b-4fbc-858b-1648ec6124e9","Type":"ContainerDied","Data":"4d16c36ae9966399f4ecb830e5f8cae16ea432d05d8e6667be57fd584b7641d0"}
Nov 01 05:53:16.693609 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:16.693551    5689 scope.go:122] "RemoveContainer" containerID="7ae3b46ccabc42915dfdc51dafc01429fa1d2b7fb7e917fe70904e176d03fb08"
Nov 01 05:53:16.694900 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:16.694878    5689 generic.go:356] "Generic (PLEG): container finished" podID="75700ead-89a9-4ee7-a12a-934b6c5cb2cb" containerID="535e7da13a60fc5686a3870e965895709a6a154b689ef823b37d7ebadca43938" exitCode=0
Nov 01 05:53:16.695005 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:16.694921    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-cpzlj"
Nov 01 05:53:16.695005 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:16.694971    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-cpzlj" event={"ID":"75700ead-89a9-4ee7-a12a-934b6c5cb2cb","Type":"ContainerDied","Data":"535e7da13a60fc5686a3870e965895709a6a154b689ef823b37d7ebadca43938"}
Nov 01 05:53:16.695110 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:16.695021    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-cpzlj" event={"ID":"75700ead-89a9-4ee7-a12a-934b6c5cb2cb","Type":"ContainerDied","Data":"6a9df6c0f28a02796c3f3644a2ebfd8eade450655625ad94538a8c6a779ab826"}
Nov 01 05:53:16.706671 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:16.706652    5689 scope.go:122] "RemoveContainer" containerID="7ae3b46ccabc42915dfdc51dafc01429fa1d2b7fb7e917fe70904e176d03fb08"
Nov 01 05:53:16.706980 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:53:16.706952    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ae3b46ccabc42915dfdc51dafc01429fa1d2b7fb7e917fe70904e176d03fb08\": container with ID starting with 7ae3b46ccabc42915dfdc51dafc01429fa1d2b7fb7e917fe70904e176d03fb08 not found: ID does not exist" containerID="7ae3b46ccabc42915dfdc51dafc01429fa1d2b7fb7e917fe70904e176d03fb08"
Nov 01 05:53:16.707059 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:16.706978    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ae3b46ccabc42915dfdc51dafc01429fa1d2b7fb7e917fe70904e176d03fb08"} err="failed to get container status \"7ae3b46ccabc42915dfdc51dafc01429fa1d2b7fb7e917fe70904e176d03fb08\": rpc error: code = NotFound desc = could not find container \"7ae3b46ccabc42915dfdc51dafc01429fa1d2b7fb7e917fe70904e176d03fb08\": container with ID starting with 7ae3b46ccabc42915dfdc51dafc01429fa1d2b7fb7e917fe70904e176d03fb08 not found: ID does not exist"
Nov 01 05:53:16.707059 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:16.706995    5689 scope.go:122] "RemoveContainer" containerID="535e7da13a60fc5686a3870e965895709a6a154b689ef823b37d7ebadca43938"
Nov 01 05:53:16.716164 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:16.716145    5689 scope.go:122] "RemoveContainer" containerID="535e7da13a60fc5686a3870e965895709a6a154b689ef823b37d7ebadca43938"
Nov 01 05:53:16.716429 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:53:16.716414    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"535e7da13a60fc5686a3870e965895709a6a154b689ef823b37d7ebadca43938\": container with ID starting with 535e7da13a60fc5686a3870e965895709a6a154b689ef823b37d7ebadca43938 not found: ID does not exist" containerID="535e7da13a60fc5686a3870e965895709a6a154b689ef823b37d7ebadca43938"
Nov 01 05:53:16.716524 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:16.716439    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"535e7da13a60fc5686a3870e965895709a6a154b689ef823b37d7ebadca43938"} err="failed to get container status \"535e7da13a60fc5686a3870e965895709a6a154b689ef823b37d7ebadca43938\": rpc error: code = NotFound desc = could not find container \"535e7da13a60fc5686a3870e965895709a6a154b689ef823b37d7ebadca43938\": container with ID starting with 535e7da13a60fc5686a3870e965895709a6a154b689ef823b37d7ebadca43938 not found: ID does not exist"
Nov 01 05:53:16.856947 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:16.856917    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gbhn9"]
Nov 01 05:53:16.906542 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:16.906509    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-gbhn9"]
Nov 01 05:53:17.016110 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:17.016083    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-cpzlj"]
Nov 01 05:53:17.019704 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:17.019680    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-25565930-openshift-default-64f5666649-cpzlj"]
Nov 01 05:53:18.068297 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:18.068255    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75700ead-89a9-4ee7-a12a-934b6c5cb2cb" path="/var/lib/kubelet/pods/75700ead-89a9-4ee7-a12a-934b6c5cb2cb/volumes"
Nov 01 05:53:18.068942 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:18.068920    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e67e1922-ea2b-4fbc-858b-1648ec6124e9" path="/var/lib/kubelet/pods/e67e1922-ea2b-4fbc-858b-1648ec6124e9/volumes"
Nov 01 05:53:18.172401 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:18.172360    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:53:18.348147 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:18.348053    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:53:18.517731 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:18.517698    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:53:18.693445 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:18.693413    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:53:20.522960 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:20.522738    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:53:21.131131 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:21.131095    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:53:21.506159 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:21.506126    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:53:23.329729 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:23.329693    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:53:25.587491 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:25.587427    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:53:25.765213 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:25.765170    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:53:25.945196 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:25.945165    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:53:28.343390 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:28.343358    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:53:30.044657 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:30.044625    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:53:30.068991 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:53:30.068955    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:53:32.369958 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:32.369926    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:53:32.552608 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:32.552561    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:53:33.470874 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:33.470838    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:53:38.127479 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:38.127433    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:53:38.163780 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:38.163719    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:53:39.791666 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:39.791615    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:53:40.015714 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:40.015677    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:53:40.406477 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:40.405998    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:53:43.071659 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:43.071631    5689 scope.go:122] "RemoveContainer" containerID="f05602983d23ffed79fb868010a4534e4b1733b17158fbb16b8c2d94b1ba6b5b"
Nov 01 05:53:43.103794 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:43.103771    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:53:47.501648 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:47.501608    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:53:47.891193 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:47.891120    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:53:52.607712 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:52.607680    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:53:53.545714 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:53.545681    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:53:54.790699 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:54.790669    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:53:57.475801 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:53:57.475766    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:54:00.069307 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:54:00.069272    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:54:02.120186 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:54:02.120150    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:54:04.879587 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:54:04.879557    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:54:05.064280 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:54:05.064255    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:54:06.913305 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:54:06.913267    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:54:10.885118 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:54:10.885084    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:54:11.716430 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:54:11.716399    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:54:14.253748 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:54:14.253708    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:54:16.557479 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:54:16.549553    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-c2zj8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 05:54:16.557479 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:54:16.549623    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c2zj8" podUID="da1ad4a4-629e-4ce5-a336-1bf832074db4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 05:54:16.606479 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:54:16.603552    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-vwd6x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 05:54:16.606479 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:54:16.603627    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vwd6x" podUID="5b840b67-ec8d-45e0-91b4-0a8dc7a353eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 05:54:16.660333 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:54:16.660254    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:54:21.400200 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:54:21.400168    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:54:26.211888 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:54:26.211852    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:54:30.068893 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:54:30.068859    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:54:30.300095 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:54:30.300058    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:54:30.659635 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:54:30.659604    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d96a66-7f878b74fd-m8zrw"]
Nov 01 05:54:30.660153 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:54:30.660139    5689 cpu_manager.go:414] "containerMap: removing container" podUID="e67e1922-ea2b-4fbc-858b-1648ec6124e9" containerName="istio-proxy"
Nov 01 05:54:30.660198 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:54:30.660155    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e67e1922-ea2b-4fbc-858b-1648ec6124e9" containerName="istio-proxy"
Nov 01 05:54:30.660198 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:54:30.660173    5689 cpu_manager.go:414] "containerMap: removing container" podUID="5b679696-4dae-41d8-bc1f-1a598aeedec9" containerName="payload-pre-processing"
Nov 01 05:54:30.660198 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:54:30.660179    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b679696-4dae-41d8-bc1f-1a598aeedec9" containerName="payload-pre-processing"
Nov 01 05:54:30.660198 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:54:30.660195    5689 cpu_manager.go:414] "containerMap: removing container" podUID="75700ead-89a9-4ee7-a12a-934b6c5cb2cb" containerName="istio-proxy"
Nov 01 05:54:30.660328 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:54:30.660200    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75700ead-89a9-4ee7-a12a-934b6c5cb2cb" containerName="istio-proxy"
Nov 01 05:54:30.660328 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:54:30.660221    5689 cpu_manager.go:414] "containerMap: removing container" podUID="0e6b057a-6ebd-4a97-8ef9-e2c77f0379c0" containerName="payload-processing"
Nov 01 05:54:30.660328 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:54:30.660226    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e6b057a-6ebd-4a97-8ef9-e2c77f0379c0" containerName="payload-processing"
Nov 01 05:54:30.660328 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:54:30.660308    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="e67e1922-ea2b-4fbc-858b-1648ec6124e9" containerName="istio-proxy"
Nov 01 05:54:30.660328 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:54:30.660317    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b679696-4dae-41d8-bc1f-1a598aeedec9" containerName="payload-pre-processing"
Nov 01 05:54:30.660328 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:54:30.660325    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="75700ead-89a9-4ee7-a12a-934b6c5cb2cb" containerName="istio-proxy"
Nov 01 05:54:30.660570 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:54:30.660335    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e6b057a-6ebd-4a97-8ef9-e2c77f0379c0" containerName="payload-processing"
Nov 01 05:54:30.663768 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:54:30.663751    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d96a66-7f878b74fd-m8zrw"
Nov 01 05:54:30.715952 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:54:30.715927    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-d96a66-serving-cert\""
Nov 01 05:54:30.767611 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:54:30.767580    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d96a66-7f878b74fd-m8zrw"]
Nov 01 05:54:30.770678 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:54:30.770649    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:54:30.789868 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:54:30.789832    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef1eafdc-0b49-42ed-ac95-6bddfd5486ec-maas-api-tls\") pod \"maas-api-e2e-extmod-a-d96a66-7f878b74fd-m8zrw\" (UID: \"ef1eafdc-0b49-42ed-ac95-6bddfd5486ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d96a66-7f878b74fd-m8zrw"
Nov 01 05:54:30.790007 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:54:30.789873    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff8bt\" (UniqueName: \"kubernetes.io/projected/ef1eafdc-0b49-42ed-ac95-6bddfd5486ec-kube-api-access-ff8bt\") pod \"maas-api-e2e-extmod-a-d96a66-7f878b74fd-m8zrw\" (UID: \"ef1eafdc-0b49-42ed-ac95-6bddfd5486ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d96a66-7f878b74fd-m8zrw"
Nov 01 05:54:30.790007 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:54:30.789986    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef1eafdc-0b49-42ed-ac95-6bddfd5486ec-metrics-certs\") pod \"maas-api-e2e-extmod-a-d96a66-7f878b74fd-m8zrw\" (UID: \"ef1eafdc-0b49-42ed-ac95-6bddfd5486ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d96a66-7f878b74fd-m8zrw"
Nov 01 05:54:30.790126 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:54:30.790093    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef1eafdc-0b49-42ed-ac95-6bddfd5486ec-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-d96a66-7f878b74fd-m8zrw\" (UID: \"ef1eafdc-0b49-42ed-ac95-6bddfd5486ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d96a66-7f878b74fd-m8zrw"
Nov 01 05:54:30.891174 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:54:30.891143    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef1eafdc-0b49-42ed-ac95-6bddfd5486ec-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-d96a66-7f878b74fd-m8zrw\" (UID: \"ef1eafdc-0b49-42ed-ac95-6bddfd5486ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d96a66-7f878b74fd-m8zrw"
Nov 01 05:54:30.891340 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:54:30.891255    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef1eafdc-0b49-42ed-ac95-6bddfd5486ec-maas-api-tls\") pod \"maas-api-e2e-extmod-a-d96a66-7f878b74fd-m8zrw\" (UID: \"ef1eafdc-0b49-42ed-ac95-6bddfd5486ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d96a66-7f878b74fd-m8zrw"
Nov 01 05:54:30.891340 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:54:30.891279    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff8bt\" (UniqueName: \"kubernetes.io/projected/ef1eafdc-0b49-42ed-ac95-6bddfd5486ec-kube-api-access-ff8bt\") pod \"maas-api-e2e-extmod-a-d96a66-7f878b74fd-m8zrw\" (UID: \"ef1eafdc-0b49-42ed-ac95-6bddfd5486ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d96a66-7f878b74fd-m8zrw"
Nov 01 05:54:30.891340 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:54:30.891327    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef1eafdc-0b49-42ed-ac95-6bddfd5486ec-metrics-certs\") pod \"maas-api-e2e-extmod-a-d96a66-7f878b74fd-m8zrw\" (UID: \"ef1eafdc-0b49-42ed-ac95-6bddfd5486ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d96a66-7f878b74fd-m8zrw"
Nov 01 05:54:30.891554 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:54:30.891407    5689 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-d96a66-serving-cert: secret "maas-api-e2e-extmod-a-d96a66-serving-cert" not found
Nov 01 05:54:30.891554 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:54:30.891500    5689 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef1eafdc-0b49-42ed-ac95-6bddfd5486ec-maas-api-tls podName:ef1eafdc-0b49-42ed-ac95-6bddfd5486ec nodeName:}" failed. No retries permitted until 2026-11-01 05:54:31.391479416 +0000 UTC m=+3049.462276127 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ef1eafdc-0b49-42ed-ac95-6bddfd5486ec-maas-api-tls") pod "maas-api-e2e-extmod-a-d96a66-7f878b74fd-m8zrw" (UID: "ef1eafdc-0b49-42ed-ac95-6bddfd5486ec") : secret "maas-api-e2e-extmod-a-d96a66-serving-cert" not found
Nov 01 05:54:30.892091 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:54:30.892034    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef1eafdc-0b49-42ed-ac95-6bddfd5486ec-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-d96a66-7f878b74fd-m8zrw\" (UID: \"ef1eafdc-0b49-42ed-ac95-6bddfd5486ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d96a66-7f878b74fd-m8zrw"
Nov 01 05:54:30.894166 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:54:30.894143    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef1eafdc-0b49-42ed-ac95-6bddfd5486ec-metrics-certs\") pod \"maas-api-e2e-extmod-a-d96a66-7f878b74fd-m8zrw\" (UID: \"ef1eafdc-0b49-42ed-ac95-6bddfd5486ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d96a66-7f878b74fd-m8zrw"
Nov 01 05:54:30.954206 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:54:30.954142    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:54:30.956856 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:54:30.956823    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff8bt\" (UniqueName: \"kubernetes.io/projected/ef1eafdc-0b49-42ed-ac95-6bddfd5486ec-kube-api-access-ff8bt\") pod \"maas-api-e2e-extmod-a-d96a66-7f878b74fd-m8zrw\" (UID: \"ef1eafdc-0b49-42ed-ac95-6bddfd5486ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d96a66-7f878b74fd-m8zrw"
Nov 01 05:54:31.133857 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:54:31.133817    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:54:31.404724 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:54:31.404688    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef1eafdc-0b49-42ed-ac95-6bddfd5486ec-maas-api-tls\") pod \"maas-api-e2e-extmod-a-d96a66-7f878b74fd-m8zrw\" (UID: \"ef1eafdc-0b49-42ed-ac95-6bddfd5486ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d96a66-7f878b74fd-m8zrw"
Nov 01 05:54:31.407174 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:54:31.407149    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef1eafdc-0b49-42ed-ac95-6bddfd5486ec-maas-api-tls\") pod \"maas-api-e2e-extmod-a-d96a66-7f878b74fd-m8zrw\" (UID: \"ef1eafdc-0b49-42ed-ac95-6bddfd5486ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d96a66-7f878b74fd-m8zrw"
Nov 01 05:54:31.578593 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:54:31.578557    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d96a66-7f878b74fd-m8zrw"
Nov 01 05:54:31.958036 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:54:31.958013    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d96a66-7f878b74fd-m8zrw"]
Nov 01 05:54:31.960583 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:54:31.960557    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef1eafdc_0b49_42ed_ac95_6bddfd5486ec.slice/crio-233b55f52e96ca4241349e34d337ffac2a37ac2e5b02f2bcb9d1719848dcdb71 WatchSource:0}: Error finding container 233b55f52e96ca4241349e34d337ffac2a37ac2e5b02f2bcb9d1719848dcdb71: Status 404 returned error can't find the container with id 233b55f52e96ca4241349e34d337ffac2a37ac2e5b02f2bcb9d1719848dcdb71
Nov 01 05:54:31.961994 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:54:31.961972    5689 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 05:54:32.439698 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:54:32.439666    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d96a66-7f878b74fd-m8zrw" event={"ID":"ef1eafdc-0b49-42ed-ac95-6bddfd5486ec","Type":"ContainerStarted","Data":"233b55f52e96ca4241349e34d337ffac2a37ac2e5b02f2bcb9d1719848dcdb71"}
Nov 01 05:54:33.446058 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:54:33.446024    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d96a66-7f878b74fd-m8zrw" event={"ID":"ef1eafdc-0b49-42ed-ac95-6bddfd5486ec","Type":"ContainerStarted","Data":"59557d026bc8d3faa1682b2f60b8990872b98cf1f2c18db6215327fd4336c505"}
Nov 01 05:54:33.446487 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:54:33.446101    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d96a66-7f878b74fd-m8zrw"
Nov 01 05:54:33.562442 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:54:33.562386    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d96a66-7f878b74fd-m8zrw" podStartSLOduration=2.697564757 podStartE2EDuration="3.562371273s" totalImagesPullingTime="864.806516ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:54:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:54:33.561926539 +0000 UTC m=+3051.632723260" watchObservedRunningTime="2026-11-01 05:54:33.562371273 +0000 UTC m=+3051.633167995"
Nov 01 05:54:34.939216 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:54:34.939182    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:54:35.788472 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:54:35.788424    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:54:35.970815 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:54:35.970782    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:54:39.457676 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:54:39.457647    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d96a66-7f878b74fd-m8zrw"
Nov 01 05:54:40.574963 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:54:40.574926    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:54:45.295305 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:54:45.295274    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:54:50.043516 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:54:50.043412    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:54:54.809074 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:54:54.809039    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:54:59.585744 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:54:59.585702    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:55:00.070471 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:55:00.070416    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:55:04.308468 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:55:04.308434    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:55:09.049226 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:55:09.049190    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:55:11.233964 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:55:11.233930    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:55:11.308624 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:55:11.308594    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:55:11.403946 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:55:11.403919    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:55:13.269193 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:55:13.269157    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:55:13.833270 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:55:13.833237    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:55:14.476513 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:55:14.476478    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:55:14.656000 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:55:14.655958    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:55:16.664541 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:55:16.664515    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:55:18.898832 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:55:18.898797    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:55:19.286974 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:55:19.286870    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:55:23.716490 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:55:23.716131    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:55:27.829200 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:55:27.829171    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:55:29.192203 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:55:29.192169    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:55:29.611199 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:55:29.611111    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:55:30.070530 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:55:30.070495    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:55:33.791921 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:55:33.791888    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:55:34.029639 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:55:34.029606    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:55:34.506853 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:55:34.506818    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:55:35.313952 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:55:35.313915    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:55:35.493204 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:55:35.493171    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:55:35.996535 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:55:35.996504    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:55:36.559518 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:55:36.559489    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:55:37.152161 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:55:37.152124    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:55:37.491808 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:55:37.491706    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:55:39.411728 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:55:39.411684    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:55:39.596742 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:55:39.596702    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:55:39.845330 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:55:39.845217    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:55:43.527931 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:55:43.527902    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:55:44.295411 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:55:44.295374    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:55:44.820745 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:55:44.820710    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:55:45.004323 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:55:45.004293    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:55:45.904924 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:55:45.904890    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:55:46.108104 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:55:46.108073    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:55:49.454755 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:55:49.454723    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:55:49.786700 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:55:49.786614    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:55:49.975582 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:55:49.975553    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:55:50.164865 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:55:50.164835    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:55:53.266676 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:55:53.266635    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:55:53.464399 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:55:53.464369    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:55:55.338594 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:55:55.338553    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:56:00.060070 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:56:00.060037    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:56:00.133204 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:56:00.133162    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:56:04.272639 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:56:04.272578    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:56:04.454676 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:56:04.454645    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:56:04.725336 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:56:04.725307    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:56:04.907132 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:56:04.907098    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:56:09.515222 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:56:09.515187    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:56:09.869120 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:56:09.869034    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:56:11.579138 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:56:11.579098    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:56:13.333340 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:56:13.333302    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:56:14.217838 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:56:14.217803    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:56:19.005678 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:56:19.005644    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:56:21.313414 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:56:21.313342    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:56:23.798052 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:56:23.791230    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:56:28.519040 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:56:28.519004    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:56:30.071042 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:56:30.071008    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:56:35.086806 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:56:35.086771    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:56:38.109077 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:56:38.109016    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:56:42.836619 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:56:42.836580    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:56:46.876431 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:56:46.876368    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:56:47.341865 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:56:47.341833    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:56:47.666944 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:56:47.666909    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:56:47.958752 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:56:47.958651    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:56:48.285606 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:56:48.285525    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:56:52.491754 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:56:52.491719    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:56:52.678276 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:56:52.678239    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:56:53.009263 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:56:53.009231    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:56:53.184210 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:56:53.184168    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:56:53.367223 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:56:53.367148    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:56:55.707542 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:56:55.707509    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:56:56.276573 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:56:56.276541    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:56:56.462760 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:56:56.462728    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:56:57.447809 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:56:57.447775    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:56:57.627264 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:56:57.627231    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:56:57.830168 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:56:57.830078    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:56:59.694584 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:56:59.694545    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:56:59.876943 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:56:59.876888    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:57:00.069972 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:57:00.069878    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:57:03.000892 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:03.000862    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:57:07.726901 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:07.726866    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:57:10.588960 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:10.588925    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:57:10.896670 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:10.896597    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:57:11.931792 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:11.931763    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:57:12.500325 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:12.500290    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:57:14.175302 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:14.175271    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:57:17.324298 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:17.324261    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:57:22.110970 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:22.110935    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:57:26.912988 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:26.912949    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:57:30.070477 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:57:30.070421    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:57:31.809286 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:31.809218    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:57:32.156476 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:32.156418    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:57:32.179293 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:32.179268    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx"]
Nov 01 05:57:32.179513 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:32.179490    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx" podUID="9cc52d9e-b04b-479b-b128-ece05b493ae9" containerName="istio-proxy" containerID="cri-o://75d2cc387aff6a8cfc278d5182c9ba395983507f59b2d55722e4d4bdabfb26b0" gracePeriod=30
Nov 01 05:57:32.228531 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:32.228504    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc"]
Nov 01 05:57:32.228700 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:32.228682    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc" podUID="5b2091c5-03e9-45a0-94d3-3946bcd2db7a" containerName="istio-proxy" containerID="cri-o://0a138be10c9fc7d89093b455c384b90188907d0c73769aee5b0ea14e73a1e7e9" gracePeriod=30
Nov 01 05:57:36.528886 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:36.528854    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:57:36.761050 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:36.761014    5689 patch_prober.go:29] interesting pod/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 05:57:36.761216 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:36.761076    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc" podUID="5b2091c5-03e9-45a0-94d3-3946bcd2db7a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 05:57:36.977448 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:36.977405    5689 patch_prober.go:29] interesting pod/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 05:57:36.977633 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:36.977476    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx" podUID="9cc52d9e-b04b-479b-b128-ece05b493ae9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 05:57:37.506800 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:37.506741    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx"
Nov 01 05:57:37.540757 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:37.540735    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc"
Nov 01 05:57:37.600083 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:37.600053    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cc52d9e-b04b-479b-b128-ece05b493ae9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cc52d9e-b04b-479b-b128-ece05b493ae9-workload-certs\") pod \"9cc52d9e-b04b-479b-b128-ece05b493ae9\" (UID: \"9cc52d9e-b04b-479b-b128-ece05b493ae9\") "
Nov 01 05:57:37.600212 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:37.600126    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cc52d9e-b04b-479b-b128-ece05b493ae9-istio-token\" (UniqueName: \"kubernetes.io/projected/9cc52d9e-b04b-479b-b128-ece05b493ae9-istio-token\") pod \"9cc52d9e-b04b-479b-b128-ece05b493ae9\" (UID: \"9cc52d9e-b04b-479b-b128-ece05b493ae9\") "
Nov 01 05:57:37.600212 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:37.600148    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cc52d9e-b04b-479b-b128-ece05b493ae9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cc52d9e-b04b-479b-b128-ece05b493ae9-credential-socket\") pod \"9cc52d9e-b04b-479b-b128-ece05b493ae9\" (UID: \"9cc52d9e-b04b-479b-b128-ece05b493ae9\") "
Nov 01 05:57:37.600212 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:37.600194    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cc52d9e-b04b-479b-b128-ece05b493ae9-kube-api-access-jzgcq\" (UniqueName: \"kubernetes.io/projected/9cc52d9e-b04b-479b-b128-ece05b493ae9-kube-api-access-jzgcq\") pod \"9cc52d9e-b04b-479b-b128-ece05b493ae9\" (UID: \"9cc52d9e-b04b-479b-b128-ece05b493ae9\") "
Nov 01 05:57:37.600333 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:37.600309    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9cc52d9e-b04b-479b-b128-ece05b493ae9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cc52d9e-b04b-479b-b128-ece05b493ae9-istio-podinfo\") pod \"9cc52d9e-b04b-479b-b128-ece05b493ae9\" (UID: \"9cc52d9e-b04b-479b-b128-ece05b493ae9\") "
Nov 01 05:57:37.600443 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:37.600310    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cc52d9e-b04b-479b-b128-ece05b493ae9-workload-certs" pod "9cc52d9e-b04b-479b-b128-ece05b493ae9" (UID: "9cc52d9e-b04b-479b-b128-ece05b493ae9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:57:37.600609 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:37.600470    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cc52d9e-b04b-479b-b128-ece05b493ae9-credential-socket" pod "9cc52d9e-b04b-479b-b128-ece05b493ae9" (UID: "9cc52d9e-b04b-479b-b128-ece05b493ae9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:57:37.600609 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:37.600508    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cc52d9e-b04b-479b-b128-ece05b493ae9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cc52d9e-b04b-479b-b128-ece05b493ae9-workload-socket\") pod \"9cc52d9e-b04b-479b-b128-ece05b493ae9\" (UID: \"9cc52d9e-b04b-479b-b128-ece05b493ae9\") "
Nov 01 05:57:37.600609 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:37.600545    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cc52d9e-b04b-479b-b128-ece05b493ae9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cc52d9e-b04b-479b-b128-ece05b493ae9-istio-envoy\") pod \"9cc52d9e-b04b-479b-b128-ece05b493ae9\" (UID: \"9cc52d9e-b04b-479b-b128-ece05b493ae9\") "
Nov 01 05:57:37.600736 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:37.600673    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cc52d9e-b04b-479b-b128-ece05b493ae9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cc52d9e-b04b-479b-b128-ece05b493ae9-istio-data\") pod \"9cc52d9e-b04b-479b-b128-ece05b493ae9\" (UID: \"9cc52d9e-b04b-479b-b128-ece05b493ae9\") "
Nov 01 05:57:37.600736 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:37.600722    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cc52d9e-b04b-479b-b128-ece05b493ae9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cc52d9e-b04b-479b-b128-ece05b493ae9-istiod-ca-cert\") pod \"9cc52d9e-b04b-479b-b128-ece05b493ae9\" (UID: \"9cc52d9e-b04b-479b-b128-ece05b493ae9\") "
Nov 01 05:57:37.600852 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:37.600791    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cc52d9e-b04b-479b-b128-ece05b493ae9-workload-socket" pod "9cc52d9e-b04b-479b-b128-ece05b493ae9" (UID: "9cc52d9e-b04b-479b-b128-ece05b493ae9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:57:37.600973 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:37.600942    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cc52d9e-b04b-479b-b128-ece05b493ae9-istio-data" pod "9cc52d9e-b04b-479b-b128-ece05b493ae9" (UID: "9cc52d9e-b04b-479b-b128-ece05b493ae9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:57:37.601412 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:37.601386    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cc52d9e-b04b-479b-b128-ece05b493ae9-istiod-ca-cert" pod "9cc52d9e-b04b-479b-b128-ece05b493ae9" (UID: "9cc52d9e-b04b-479b-b128-ece05b493ae9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:57:37.601556 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:37.601413    5689 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cc52d9e-b04b-479b-b128-ece05b493ae9-istio-data\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:57:37.601556 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:37.601444    5689 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cc52d9e-b04b-479b-b128-ece05b493ae9-workload-certs\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:57:37.601556 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:37.601490    5689 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cc52d9e-b04b-479b-b128-ece05b493ae9-credential-socket\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:57:37.601556 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:37.601506    5689 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cc52d9e-b04b-479b-b128-ece05b493ae9-workload-socket\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:57:37.602636 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:37.602608    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9cc52d9e-b04b-479b-b128-ece05b493ae9-istio-podinfo" pod "9cc52d9e-b04b-479b-b128-ece05b493ae9" (UID: "9cc52d9e-b04b-479b-b128-ece05b493ae9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 05:57:37.602741 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:37.602647    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cc52d9e-b04b-479b-b128-ece05b493ae9-kube-api-access-jzgcq" pod "9cc52d9e-b04b-479b-b128-ece05b493ae9" (UID: "9cc52d9e-b04b-479b-b128-ece05b493ae9"). InnerVolumeSpecName "kube-api-access-jzgcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:57:37.602741 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:37.602713    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cc52d9e-b04b-479b-b128-ece05b493ae9-istio-envoy" pod "9cc52d9e-b04b-479b-b128-ece05b493ae9" (UID: "9cc52d9e-b04b-479b-b128-ece05b493ae9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:57:37.602833 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:37.602777    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cc52d9e-b04b-479b-b128-ece05b493ae9-istio-token" pod "9cc52d9e-b04b-479b-b128-ece05b493ae9" (UID: "9cc52d9e-b04b-479b-b128-ece05b493ae9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:57:37.689016 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:37.688983    5689 generic.go:356] "Generic (PLEG): container finished" podID="9cc52d9e-b04b-479b-b128-ece05b493ae9" containerID="75d2cc387aff6a8cfc278d5182c9ba395983507f59b2d55722e4d4bdabfb26b0" exitCode=0
Nov 01 05:57:37.689198 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:37.689033    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx"
Nov 01 05:57:37.689198 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:37.689066    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx" event={"ID":"9cc52d9e-b04b-479b-b128-ece05b493ae9","Type":"ContainerDied","Data":"75d2cc387aff6a8cfc278d5182c9ba395983507f59b2d55722e4d4bdabfb26b0"}
Nov 01 05:57:37.689198 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:37.689125    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx" event={"ID":"9cc52d9e-b04b-479b-b128-ece05b493ae9","Type":"ContainerDied","Data":"65a074e1c635a58dcc2fe4b03280d6d93214b7b6adcbd0ab23de301c3cd31c83"}
Nov 01 05:57:37.689198 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:37.689147    5689 scope.go:122] "RemoveContainer" containerID="75d2cc387aff6a8cfc278d5182c9ba395983507f59b2d55722e4d4bdabfb26b0"
Nov 01 05:57:37.691518 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:37.691367    5689 generic.go:356] "Generic (PLEG): container finished" podID="5b2091c5-03e9-45a0-94d3-3946bcd2db7a" containerID="0a138be10c9fc7d89093b455c384b90188907d0c73769aee5b0ea14e73a1e7e9" exitCode=0
Nov 01 05:57:37.691765 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:37.691735    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc" event={"ID":"5b2091c5-03e9-45a0-94d3-3946bcd2db7a","Type":"ContainerDied","Data":"0a138be10c9fc7d89093b455c384b90188907d0c73769aee5b0ea14e73a1e7e9"}
Nov 01 05:57:37.691887 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:37.691788    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc" event={"ID":"5b2091c5-03e9-45a0-94d3-3946bcd2db7a","Type":"ContainerDied","Data":"0ff7796c97ba9bb5b00a012a0625cf43c6ed2461170b12424baa77bf66f6a424"}
Nov 01 05:57:37.691887 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:37.691865    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc"
Nov 01 05:57:37.701954 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:37.701935    5689 scope.go:122] "RemoveContainer" containerID="75d2cc387aff6a8cfc278d5182c9ba395983507f59b2d55722e4d4bdabfb26b0"
Nov 01 05:57:37.702257 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:57:37.702238    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75d2cc387aff6a8cfc278d5182c9ba395983507f59b2d55722e4d4bdabfb26b0\": container with ID starting with 75d2cc387aff6a8cfc278d5182c9ba395983507f59b2d55722e4d4bdabfb26b0 not found: ID does not exist" containerID="75d2cc387aff6a8cfc278d5182c9ba395983507f59b2d55722e4d4bdabfb26b0"
Nov 01 05:57:37.702334 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:37.702256    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b2091c5-03e9-45a0-94d3-3946bcd2db7a-istio-token\" (UniqueName: \"kubernetes.io/projected/5b2091c5-03e9-45a0-94d3-3946bcd2db7a-istio-token\") pod \"5b2091c5-03e9-45a0-94d3-3946bcd2db7a\" (UID: \"5b2091c5-03e9-45a0-94d3-3946bcd2db7a\") "
Nov 01 05:57:37.702334 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:37.702264    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75d2cc387aff6a8cfc278d5182c9ba395983507f59b2d55722e4d4bdabfb26b0"} err="failed to get container status \"75d2cc387aff6a8cfc278d5182c9ba395983507f59b2d55722e4d4bdabfb26b0\": rpc error: code = NotFound desc = could not find container \"75d2cc387aff6a8cfc278d5182c9ba395983507f59b2d55722e4d4bdabfb26b0\": container with ID starting with 75d2cc387aff6a8cfc278d5182c9ba395983507f59b2d55722e4d4bdabfb26b0 not found: ID does not exist"
Nov 01 05:57:37.702334 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:37.702282    5689 scope.go:122] "RemoveContainer" containerID="0a138be10c9fc7d89093b455c384b90188907d0c73769aee5b0ea14e73a1e7e9"
Nov 01 05:57:37.703734 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:37.702330    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b2091c5-03e9-45a0-94d3-3946bcd2db7a-kube-api-access-fl42b\" (UniqueName: \"kubernetes.io/projected/5b2091c5-03e9-45a0-94d3-3946bcd2db7a-kube-api-access-fl42b\") pod \"5b2091c5-03e9-45a0-94d3-3946bcd2db7a\" (UID: \"5b2091c5-03e9-45a0-94d3-3946bcd2db7a\") "
Nov 01 05:57:37.703734 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:37.702382    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b2091c5-03e9-45a0-94d3-3946bcd2db7a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b2091c5-03e9-45a0-94d3-3946bcd2db7a-credential-socket\") pod \"5b2091c5-03e9-45a0-94d3-3946bcd2db7a\" (UID: \"5b2091c5-03e9-45a0-94d3-3946bcd2db7a\") "
Nov 01 05:57:37.703734 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:37.702415    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5b2091c5-03e9-45a0-94d3-3946bcd2db7a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b2091c5-03e9-45a0-94d3-3946bcd2db7a-istio-podinfo\") pod \"5b2091c5-03e9-45a0-94d3-3946bcd2db7a\" (UID: \"5b2091c5-03e9-45a0-94d3-3946bcd2db7a\") "
Nov 01 05:57:37.703734 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:37.702508    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b2091c5-03e9-45a0-94d3-3946bcd2db7a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b2091c5-03e9-45a0-94d3-3946bcd2db7a-istio-data\") pod \"5b2091c5-03e9-45a0-94d3-3946bcd2db7a\" (UID: \"5b2091c5-03e9-45a0-94d3-3946bcd2db7a\") "
Nov 01 05:57:37.703734 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:37.702592    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b2091c5-03e9-45a0-94d3-3946bcd2db7a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b2091c5-03e9-45a0-94d3-3946bcd2db7a-workload-socket\") pod \"5b2091c5-03e9-45a0-94d3-3946bcd2db7a\" (UID: \"5b2091c5-03e9-45a0-94d3-3946bcd2db7a\") "
Nov 01 05:57:37.703734 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:37.702686    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b2091c5-03e9-45a0-94d3-3946bcd2db7a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b2091c5-03e9-45a0-94d3-3946bcd2db7a-istiod-ca-cert\") pod \"5b2091c5-03e9-45a0-94d3-3946bcd2db7a\" (UID: \"5b2091c5-03e9-45a0-94d3-3946bcd2db7a\") "
Nov 01 05:57:37.703734 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:37.702717    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b2091c5-03e9-45a0-94d3-3946bcd2db7a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b2091c5-03e9-45a0-94d3-3946bcd2db7a-istio-envoy\") pod \"5b2091c5-03e9-45a0-94d3-3946bcd2db7a\" (UID: \"5b2091c5-03e9-45a0-94d3-3946bcd2db7a\") "
Nov 01 05:57:37.703734 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:37.702750    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b2091c5-03e9-45a0-94d3-3946bcd2db7a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b2091c5-03e9-45a0-94d3-3946bcd2db7a-workload-certs\") pod \"5b2091c5-03e9-45a0-94d3-3946bcd2db7a\" (UID: \"5b2091c5-03e9-45a0-94d3-3946bcd2db7a\") "
Nov 01 05:57:37.703734 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:37.703016    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b2091c5-03e9-45a0-94d3-3946bcd2db7a-workload-socket" pod "5b2091c5-03e9-45a0-94d3-3946bcd2db7a" (UID: "5b2091c5-03e9-45a0-94d3-3946bcd2db7a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:57:37.703734 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:37.703150    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b2091c5-03e9-45a0-94d3-3946bcd2db7a-credential-socket" pod "5b2091c5-03e9-45a0-94d3-3946bcd2db7a" (UID: "5b2091c5-03e9-45a0-94d3-3946bcd2db7a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:57:37.703734 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:37.703253    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b2091c5-03e9-45a0-94d3-3946bcd2db7a-workload-certs" pod "5b2091c5-03e9-45a0-94d3-3946bcd2db7a" (UID: "5b2091c5-03e9-45a0-94d3-3946bcd2db7a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:57:37.703734 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:37.703363    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b2091c5-03e9-45a0-94d3-3946bcd2db7a-istio-data" pod "5b2091c5-03e9-45a0-94d3-3946bcd2db7a" (UID: "5b2091c5-03e9-45a0-94d3-3946bcd2db7a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:57:37.703734 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:37.703526    5689 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b2091c5-03e9-45a0-94d3-3946bcd2db7a-istio-data\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:57:37.703734 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:37.703548    5689 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cc52d9e-b04b-479b-b128-ece05b493ae9-istiod-ca-cert\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:57:37.703734 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:37.703565    5689 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b2091c5-03e9-45a0-94d3-3946bcd2db7a-workload-socket\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:57:37.703734 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:37.703580    5689 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cc52d9e-b04b-479b-b128-ece05b493ae9-istio-token\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:57:37.703734 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:37.703591    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b2091c5-03e9-45a0-94d3-3946bcd2db7a-istiod-ca-cert" pod "5b2091c5-03e9-45a0-94d3-3946bcd2db7a" (UID: "5b2091c5-03e9-45a0-94d3-3946bcd2db7a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:57:37.703734 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:37.703595    5689 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b2091c5-03e9-45a0-94d3-3946bcd2db7a-workload-certs\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:57:37.703734 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:37.703633    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzgcq\" (UniqueName: \"kubernetes.io/projected/9cc52d9e-b04b-479b-b128-ece05b493ae9-kube-api-access-jzgcq\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:57:37.703734 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:37.703649    5689 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cc52d9e-b04b-479b-b128-ece05b493ae9-istio-podinfo\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:57:37.703734 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:37.703665    5689 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cc52d9e-b04b-479b-b128-ece05b493ae9-istio-envoy\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:57:37.703734 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:37.703679    5689 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b2091c5-03e9-45a0-94d3-3946bcd2db7a-credential-socket\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:57:37.705005 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:37.704983    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b2091c5-03e9-45a0-94d3-3946bcd2db7a-istio-token" pod "5b2091c5-03e9-45a0-94d3-3946bcd2db7a" (UID: "5b2091c5-03e9-45a0-94d3-3946bcd2db7a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:57:37.705217 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:37.705198    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5b2091c5-03e9-45a0-94d3-3946bcd2db7a-istio-podinfo" pod "5b2091c5-03e9-45a0-94d3-3946bcd2db7a" (UID: "5b2091c5-03e9-45a0-94d3-3946bcd2db7a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 05:57:37.705552 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:37.705534    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b2091c5-03e9-45a0-94d3-3946bcd2db7a-kube-api-access-fl42b" pod "5b2091c5-03e9-45a0-94d3-3946bcd2db7a" (UID: "5b2091c5-03e9-45a0-94d3-3946bcd2db7a"). InnerVolumeSpecName "kube-api-access-fl42b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:57:37.706337 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:37.706319    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b2091c5-03e9-45a0-94d3-3946bcd2db7a-istio-envoy" pod "5b2091c5-03e9-45a0-94d3-3946bcd2db7a" (UID: "5b2091c5-03e9-45a0-94d3-3946bcd2db7a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:57:37.714050 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:37.714027    5689 scope.go:122] "RemoveContainer" containerID="0a138be10c9fc7d89093b455c384b90188907d0c73769aee5b0ea14e73a1e7e9"
Nov 01 05:57:37.714392 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:57:37.714372    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a138be10c9fc7d89093b455c384b90188907d0c73769aee5b0ea14e73a1e7e9\": container with ID starting with 0a138be10c9fc7d89093b455c384b90188907d0c73769aee5b0ea14e73a1e7e9 not found: ID does not exist" containerID="0a138be10c9fc7d89093b455c384b90188907d0c73769aee5b0ea14e73a1e7e9"
Nov 01 05:57:37.714447 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:37.714402    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a138be10c9fc7d89093b455c384b90188907d0c73769aee5b0ea14e73a1e7e9"} err="failed to get container status \"0a138be10c9fc7d89093b455c384b90188907d0c73769aee5b0ea14e73a1e7e9\": rpc error: code = NotFound desc = could not find container \"0a138be10c9fc7d89093b455c384b90188907d0c73769aee5b0ea14e73a1e7e9\": container with ID starting with 0a138be10c9fc7d89093b455c384b90188907d0c73769aee5b0ea14e73a1e7e9 not found: ID does not exist"
Nov 01 05:57:37.804430 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:37.804366    5689 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b2091c5-03e9-45a0-94d3-3946bcd2db7a-istio-token\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:57:37.804430 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:37.804393    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fl42b\" (UniqueName: \"kubernetes.io/projected/5b2091c5-03e9-45a0-94d3-3946bcd2db7a-kube-api-access-fl42b\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:57:37.804430 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:37.804405    5689 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b2091c5-03e9-45a0-94d3-3946bcd2db7a-istio-podinfo\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:57:37.804430 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:37.804414    5689 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b2091c5-03e9-45a0-94d3-3946bcd2db7a-istiod-ca-cert\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:57:37.804430 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:37.804424    5689 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b2091c5-03e9-45a0-94d3-3946bcd2db7a-istio-envoy\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:57:37.915775 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:37.915739    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx"]
Nov 01 05:57:37.918208 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:37.918176    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jm8wx"]
Nov 01 05:57:38.069070 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:38.069004    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cc52d9e-b04b-479b-b128-ece05b493ae9" path="/var/lib/kubelet/pods/9cc52d9e-b04b-479b-b128-ece05b493ae9/volumes"
Nov 01 05:57:38.113499 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:38.113445    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc"]
Nov 01 05:57:38.117039 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:38.117003    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-7a360a-openshift-default-5dd9fb679c-jnbcc"]
Nov 01 05:57:38.138724 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:38.138699    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:57:40.068011 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:40.067969    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b2091c5-03e9-45a0-94d3-3946bcd2db7a" path="/var/lib/kubelet/pods/5b2091c5-03e9-45a0-94d3-3946bcd2db7a/volumes"
Nov 01 05:57:41.315636 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:41.315601    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:57:46.028611 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:46.028576    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:57:46.511689 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:46.511658    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:57:50.796476 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:50.796355    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:57:55.603013 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:57:55.602970    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:58:00.025158 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:58:00.025121    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-897c87-588d6567c7-7nvdj"]
Nov 01 05:58:00.025716 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:58:00.025700    5689 cpu_manager.go:414] "containerMap: removing container" podUID="9cc52d9e-b04b-479b-b128-ece05b493ae9" containerName="istio-proxy"
Nov 01 05:58:00.025762 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:58:00.025717    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cc52d9e-b04b-479b-b128-ece05b493ae9" containerName="istio-proxy"
Nov 01 05:58:00.025762 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:58:00.025757    5689 cpu_manager.go:414] "containerMap: removing container" podUID="5b2091c5-03e9-45a0-94d3-3946bcd2db7a" containerName="istio-proxy"
Nov 01 05:58:00.025762 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:58:00.025762    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b2091c5-03e9-45a0-94d3-3946bcd2db7a" containerName="istio-proxy"
Nov 01 05:58:00.025861 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:58:00.025852    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b2091c5-03e9-45a0-94d3-3946bcd2db7a" containerName="istio-proxy"
Nov 01 05:58:00.025895 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:58:00.025865    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cc52d9e-b04b-479b-b128-ece05b493ae9" containerName="istio-proxy"
Nov 01 05:58:00.030562 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:58:00.030538    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-897c87-588d6567c7-7nvdj"
Nov 01 05:58:00.068995 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:58:00.068960    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:58:00.082509 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:58:00.082492    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-897c87-serving-cert\""
Nov 01 05:58:00.134159 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:58:00.134131    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-897c87-588d6567c7-7nvdj"]
Nov 01 05:58:00.161367 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:58:00.161332    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ls5pp\" (UniqueName: \"kubernetes.io/projected/3d1f1368-d8ef-480f-916c-27fb53b44b25-kube-api-access-ls5pp\") pod \"maas-api-e2e-shared-a-w1-897c87-588d6567c7-7nvdj\" (UID: \"3d1f1368-d8ef-480f-916c-27fb53b44b25\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-897c87-588d6567c7-7nvdj"
Nov 01 05:58:00.161562 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:58:00.161406    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d1f1368-d8ef-480f-916c-27fb53b44b25-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-897c87-588d6567c7-7nvdj\" (UID: \"3d1f1368-d8ef-480f-916c-27fb53b44b25\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-897c87-588d6567c7-7nvdj"
Nov 01 05:58:00.161562 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:58:00.161480    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d1f1368-d8ef-480f-916c-27fb53b44b25-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-897c87-588d6567c7-7nvdj\" (UID: \"3d1f1368-d8ef-480f-916c-27fb53b44b25\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-897c87-588d6567c7-7nvdj"
Nov 01 05:58:00.161562 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:58:00.161536    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d1f1368-d8ef-480f-916c-27fb53b44b25-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-897c87-588d6567c7-7nvdj\" (UID: \"3d1f1368-d8ef-480f-916c-27fb53b44b25\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-897c87-588d6567c7-7nvdj"
Nov 01 05:58:00.250936 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:58:00.250903    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:58:00.263195 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:58:00.263169    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d1f1368-d8ef-480f-916c-27fb53b44b25-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-897c87-588d6567c7-7nvdj\" (UID: \"3d1f1368-d8ef-480f-916c-27fb53b44b25\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-897c87-588d6567c7-7nvdj"
Nov 01 05:58:00.263292 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:58:00.263206    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d1f1368-d8ef-480f-916c-27fb53b44b25-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-897c87-588d6567c7-7nvdj\" (UID: \"3d1f1368-d8ef-480f-916c-27fb53b44b25\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-897c87-588d6567c7-7nvdj"
Nov 01 05:58:00.263336 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:58:00.263309    5689 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-897c87-serving-cert: secret "maas-api-e2e-shared-a-w1-897c87-serving-cert" not found
Nov 01 05:58:00.263372 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:58:00.263342    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls5pp\" (UniqueName: \"kubernetes.io/projected/3d1f1368-d8ef-480f-916c-27fb53b44b25-kube-api-access-ls5pp\") pod \"maas-api-e2e-shared-a-w1-897c87-588d6567c7-7nvdj\" (UID: \"3d1f1368-d8ef-480f-916c-27fb53b44b25\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-897c87-588d6567c7-7nvdj"
Nov 01 05:58:00.263372 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:58:00.263369    5689 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d1f1368-d8ef-480f-916c-27fb53b44b25-maas-api-tls podName:3d1f1368-d8ef-480f-916c-27fb53b44b25 nodeName:}" failed. No retries permitted until 2026-11-01 05:58:00.763355365 +0000 UTC m=+3258.834152063 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3d1f1368-d8ef-480f-916c-27fb53b44b25-maas-api-tls") pod "maas-api-e2e-shared-a-w1-897c87-588d6567c7-7nvdj" (UID: "3d1f1368-d8ef-480f-916c-27fb53b44b25") : secret "maas-api-e2e-shared-a-w1-897c87-serving-cert" not found
Nov 01 05:58:00.263476 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:58:00.263421    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d1f1368-d8ef-480f-916c-27fb53b44b25-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-897c87-588d6567c7-7nvdj\" (UID: \"3d1f1368-d8ef-480f-916c-27fb53b44b25\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-897c87-588d6567c7-7nvdj"
Nov 01 05:58:00.264065 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:58:00.263994    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d1f1368-d8ef-480f-916c-27fb53b44b25-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-897c87-588d6567c7-7nvdj\" (UID: \"3d1f1368-d8ef-480f-916c-27fb53b44b25\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-897c87-588d6567c7-7nvdj"
Nov 01 05:58:00.265880 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:58:00.265854    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d1f1368-d8ef-480f-916c-27fb53b44b25-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-897c87-588d6567c7-7nvdj\" (UID: \"3d1f1368-d8ef-480f-916c-27fb53b44b25\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-897c87-588d6567c7-7nvdj"
Nov 01 05:58:00.326217 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:58:00.326146    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls5pp\" (UniqueName: \"kubernetes.io/projected/3d1f1368-d8ef-480f-916c-27fb53b44b25-kube-api-access-ls5pp\") pod \"maas-api-e2e-shared-a-w1-897c87-588d6567c7-7nvdj\" (UID: \"3d1f1368-d8ef-480f-916c-27fb53b44b25\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-897c87-588d6567c7-7nvdj"
Nov 01 05:58:00.439567 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:58:00.435486    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:58:00.615376 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:58:00.615294    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:58:00.769263 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:58:00.769233    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d1f1368-d8ef-480f-916c-27fb53b44b25-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-897c87-588d6567c7-7nvdj\" (UID: \"3d1f1368-d8ef-480f-916c-27fb53b44b25\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-897c87-588d6567c7-7nvdj"
Nov 01 05:58:00.771772 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:58:00.771752    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d1f1368-d8ef-480f-916c-27fb53b44b25-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-897c87-588d6567c7-7nvdj\" (UID: \"3d1f1368-d8ef-480f-916c-27fb53b44b25\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-897c87-588d6567c7-7nvdj"
Nov 01 05:58:00.943583 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:58:00.943549    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-897c87-588d6567c7-7nvdj"
Nov 01 05:58:01.322129 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:58:01.322106    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-897c87-588d6567c7-7nvdj"]
Nov 01 05:58:01.324307 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:58:01.324283    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d1f1368_d8ef_480f_916c_27fb53b44b25.slice/crio-5e937fcc8cf650201de6ba1a236585fabd8b726609cd85f32015e801ae9ee057 WatchSource:0}: Error finding container 5e937fcc8cf650201de6ba1a236585fabd8b726609cd85f32015e801ae9ee057: Status 404 returned error can't find the container with id 5e937fcc8cf650201de6ba1a236585fabd8b726609cd85f32015e801ae9ee057
Nov 01 05:58:01.834525 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:58:01.834490    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-897c87-588d6567c7-7nvdj" event={"ID":"3d1f1368-d8ef-480f-916c-27fb53b44b25","Type":"ContainerStarted","Data":"5e937fcc8cf650201de6ba1a236585fabd8b726609cd85f32015e801ae9ee057"}
Nov 01 05:58:02.841719 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:58:02.841678    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-897c87-588d6567c7-7nvdj" event={"ID":"3d1f1368-d8ef-480f-916c-27fb53b44b25","Type":"ContainerStarted","Data":"780937dc013a34c53e9a9a0091368dfd84c88158a416240d078b7ee2a661589a"}
Nov 01 05:58:02.842107 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:58:02.841749    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-897c87-588d6567c7-7nvdj"
Nov 01 05:58:02.960402 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:58:02.960350    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-897c87-588d6567c7-7nvdj" podStartSLOduration=3.032926779 podStartE2EDuration="3.960336725s" totalImagesPullingTime="927.409946ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:57:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:58:02.958066426 +0000 UTC m=+3261.028863146" watchObservedRunningTime="2026-11-01 05:58:02.960336725 +0000 UTC m=+3261.031133444"
Nov 01 05:58:05.288443 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:58:05.288380    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:58:08.852071 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:58:08.852043    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-897c87-588d6567c7-7nvdj"
Nov 01 05:58:10.113757 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:58:10.113722    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:58:12.280163 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:58:12.280129    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:58:12.662936 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:58:12.662877    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:58:13.754295 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:58:13.754257    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:58:13.929212 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:58:13.929172    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:58:14.104952 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:58:14.104878    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:58:14.299834 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:58:14.299798    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:58:15.121841 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:58:15.121811    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:58:15.304142 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:58:15.304032    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:58:15.500443 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:58:15.500410    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:58:18.159159 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:58:18.159129    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:58:19.268217 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:58:19.268185    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:58:20.304235 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:58:20.304189    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:58:25.100532 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:58:25.100492    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:58:30.007484 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:58:30.007420    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:58:30.076626 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:58:30.076593    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:58:31.624956 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:58:31.624920    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:58:34.932479 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:58:34.928494    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:58:39.766608 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:58:39.766198    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:58:44.535702 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:58:44.535481    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:58:49.211006 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:58:49.210969    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:58:53.901994 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:58:53.901962    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:58:56.241229 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:58:56.241199    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:58:56.415838 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:58:56.415810    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:58:58.610098 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:58:58.610062    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:58:58.718405 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:58:58.718378    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:58:58.891513 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:58:58.891415    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:00.040926 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:00.040895    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-c7f95pfdk"]
Nov 01 05:59:00.045038 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:00.045018    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-c7f95pfdk"
Nov 01 05:59:00.060068 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:00.060043    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/eaed5d67-d647-4f94-8f6f-446d14c2d2a1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-c7f95pfdk\" (UID: \"eaed5d67-d647-4f94-8f6f-446d14c2d2a1\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-c7f95pfdk"
Nov 01 05:59:00.060190 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:00.060104    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaed5d67-d647-4f94-8f6f-446d14c2d2a1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-c7f95pfdk\" (UID: \"eaed5d67-d647-4f94-8f6f-446d14c2d2a1\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-c7f95pfdk"
Nov 01 05:59:00.060190 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:00.060119    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/eaed5d67-d647-4f94-8f6f-446d14c2d2a1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-c7f95pfdk\" (UID: \"eaed5d67-d647-4f94-8f6f-446d14c2d2a1\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-c7f95pfdk"
Nov 01 05:59:00.060333 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:00.060311    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrl9k\" (UniqueName: \"kubernetes.io/projected/eaed5d67-d647-4f94-8f6f-446d14c2d2a1-kube-api-access-mrl9k\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-c7f95pfdk\" (UID: \"eaed5d67-d647-4f94-8f6f-446d14c2d2a1\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-c7f95pfdk"
Nov 01 05:59:00.070866 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:59:00.070779    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:59:00.097173 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:00.097148    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-dee6aa\"/\"e2e-cfl-eafa67-provider-cred\""
Nov 01 05:59:00.097297 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:00.097224    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-dee6aa\"/\"routing-overlay\""
Nov 01 05:59:00.146399 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:00.146370    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-dee6aa\"/\"openshift-service-ca.crt\""
Nov 01 05:59:00.146399 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:00.146370    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-dee6aa\"/\"payload-processing-external-model-plugins-e2e-extmod-a-dee6aa\""
Nov 01 05:59:00.146683 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:00.146411    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-dee6aa\"/\"agc-cred-30604247b165\""
Nov 01 05:59:00.146683 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:00.146499    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-dee6aa\"/\"payload-processing-external-model-e2e--19b5d07b-dockercfg-rd4xw\""
Nov 01 05:59:00.146683 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:00.146614    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-dee6aa\"/\"kube-root-ca.crt\""
Nov 01 05:59:00.149193 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:00.149163    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-c7f95pfdk"]
Nov 01 05:59:00.161066 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:00.161042    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrl9k\" (UniqueName: \"kubernetes.io/projected/eaed5d67-d647-4f94-8f6f-446d14c2d2a1-kube-api-access-mrl9k\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-c7f95pfdk\" (UID: \"eaed5d67-d647-4f94-8f6f-446d14c2d2a1\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-c7f95pfdk"
Nov 01 05:59:00.161176 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:00.161158    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/eaed5d67-d647-4f94-8f6f-446d14c2d2a1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-c7f95pfdk\" (UID: \"eaed5d67-d647-4f94-8f6f-446d14c2d2a1\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-c7f95pfdk"
Nov 01 05:59:00.161271 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:00.161255    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaed5d67-d647-4f94-8f6f-446d14c2d2a1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-c7f95pfdk\" (UID: \"eaed5d67-d647-4f94-8f6f-446d14c2d2a1\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-c7f95pfdk"
Nov 01 05:59:00.161308 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:00.161286    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/eaed5d67-d647-4f94-8f6f-446d14c2d2a1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-c7f95pfdk\" (UID: \"eaed5d67-d647-4f94-8f6f-446d14c2d2a1\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-c7f95pfdk"
Nov 01 05:59:00.161990 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:00.161964    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/eaed5d67-d647-4f94-8f6f-446d14c2d2a1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-c7f95pfdk\" (UID: \"eaed5d67-d647-4f94-8f6f-446d14c2d2a1\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-c7f95pfdk"
Nov 01 05:59:00.162107 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:00.162090    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaed5d67-d647-4f94-8f6f-446d14c2d2a1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-c7f95pfdk\" (UID: \"eaed5d67-d647-4f94-8f6f-446d14c2d2a1\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-c7f95pfdk"
Nov 01 05:59:00.163981 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:00.163963    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/eaed5d67-d647-4f94-8f6f-446d14c2d2a1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-c7f95pfdk\" (UID: \"eaed5d67-d647-4f94-8f6f-446d14c2d2a1\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-c7f95pfdk"
Nov 01 05:59:00.221265 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:00.221245    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrl9k\" (UniqueName: \"kubernetes.io/projected/eaed5d67-d647-4f94-8f6f-446d14c2d2a1-kube-api-access-mrl9k\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-c7f95pfdk\" (UID: \"eaed5d67-d647-4f94-8f6f-446d14c2d2a1\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-c7f95pfdk"
Nov 01 05:59:00.358255 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:00.358186    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-c7f95pfdk"
Nov 01 05:59:00.733927 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:00.733901    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-c7f95pfdk"]
Nov 01 05:59:00.736683 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:59:00.736654    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaed5d67_d647_4f94_8f6f_446d14c2d2a1.slice/crio-389ff23d5e98be1e117fe90634bab4a8764508b679e7ff0a5d57231f53b7639b WatchSource:0}: Error finding container 389ff23d5e98be1e117fe90634bab4a8764508b679e7ff0a5d57231f53b7639b: Status 404 returned error can't find the container with id 389ff23d5e98be1e117fe90634bab4a8764508b679e7ff0a5d57231f53b7639b
Nov 01 05:59:01.275910 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:01.275878    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-c7f95pfdk" event={"ID":"eaed5d67-d647-4f94-8f6f-446d14c2d2a1","Type":"ContainerStarted","Data":"5950624539e7a47bcbad93dc44fc0542e41b0ac032a3d7f05e2fdcbb590a040b"}
Nov 01 05:59:01.276295 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:01.275934    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-c7f95pfdk" event={"ID":"eaed5d67-d647-4f94-8f6f-446d14c2d2a1","Type":"ContainerStarted","Data":"389ff23d5e98be1e117fe90634bab4a8764508b679e7ff0a5d57231f53b7639b"}
Nov 01 05:59:01.276295 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:01.276008    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-c7f95pfdk"
Nov 01 05:59:01.392286 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:01.392235    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-c7f95pfdk" podStartSLOduration=2.3922211669999998 podStartE2EDuration="2.392221167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:58:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:59:01.390818025 +0000 UTC m=+3319.461614747" watchObservedRunningTime="2026-11-01 05:59:01.392221167 +0000 UTC m=+3319.463017887"
Nov 01 05:59:02.661923 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:02.661888    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:03.512877 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:03.512796    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:07.582740 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:07.582706    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-3d70ff/distinct-2-w2-3d70ff-kserve-868b775cd4-nhx4f"]
Nov 01 05:59:07.632316 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:07.632290    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-3d70ff/simulator-w2-3d70ff-kserve-898c5fb4f-t5bzk"]
Nov 01 05:59:07.713230 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:07.713193    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:08.310341 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:08.310304    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:09.679329 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:09.679284    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:12.284960 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:12.284925    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-c7f95pfdk" podUID="eaed5d67-d647-4f94-8f6f-446d14c2d2a1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 05:59:12.619512 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:12.619387    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:12.924684 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:12.924648    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:13.133480 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:13.132997    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:15.779780 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:15.779714    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:15.953913 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:15.953877    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:18.119104 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:18.119069    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:20.432344 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:20.432265    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:20.861372 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:20.861295    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:21.372143 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:21.372111    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-c7f95pfdk"]
Nov 01 05:59:21.373525 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:21.373493    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-c7f95pfdk" podUID="eaed5d67-d647-4f94-8f6f-446d14c2d2a1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 05:59:21.560739 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:21.560703    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:21.817182 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:21.817148    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:22.604143 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:22.604108    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-3d70ff/distinct-2-w2-3d70ff-kserve-868b775cd4-nhx4f" podUID="0c79347d-eb2e-49b2-af27-48ddcf86b7bb" containerName="main" containerID="cri-o://43ab0c1f2c3c4e1ea9d4f3b52217df17a7c8d5d9aab7c8b3959f70b2c440c412" gracePeriod=45
Nov 01 05:59:22.643962 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:22.643936    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:22.652570 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:22.652543    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-3d70ff/simulator-w2-3d70ff-kserve-898c5fb4f-t5bzk" podUID="05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca" containerName="main" containerID="cri-o://eb663254760978ac1c5ae561a4582a7bfe3186f5a4e0e410679c39fb34422a64" gracePeriod=45
Nov 01 05:59:22.665128 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:22.665105    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3d70ff/distinct-2-w2-3d70ff-kserve-868b775cd4-nhx4f" podUID="0c79347d-eb2e-49b2-af27-48ddcf86b7bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": dial tcp 10.134.0.64:8000: connect: connection refused"
Nov 01 05:59:22.933947 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:22.933924    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-3d70ff/distinct-2-w2-3d70ff-kserve-868b775cd4-nhx4f"
Nov 01 05:59:22.937078 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:22.937058    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-3d70ff/simulator-w2-3d70ff-kserve-898c5fb4f-t5bzk"
Nov 01 05:59:22.995021 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:22.994993    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c79347d-eb2e-49b2-af27-48ddcf86b7bb-kube-api-access-5vrbr\" (UniqueName: \"kubernetes.io/projected/0c79347d-eb2e-49b2-af27-48ddcf86b7bb-kube-api-access-5vrbr\") pod \"0c79347d-eb2e-49b2-af27-48ddcf86b7bb\" (UID: \"0c79347d-eb2e-49b2-af27-48ddcf86b7bb\") "
Nov 01 05:59:22.995158 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:22.995077    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca-tls-certs\" (UniqueName: \"kubernetes.io/secret/05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca-tls-certs\") pod \"05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca\" (UID: \"05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca\") "
Nov 01 05:59:22.995158 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:22.995143    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca-model-cache\") pod \"05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca\" (UID: \"05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca\") "
Nov 01 05:59:22.995281 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:22.995226    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca-home\" (UniqueName: \"kubernetes.io/empty-dir/05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca-home\") pod \"05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca\" (UID: \"05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca\") "
Nov 01 05:59:22.995281 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:22.995256    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c79347d-eb2e-49b2-af27-48ddcf86b7bb-home\" (UniqueName: \"kubernetes.io/empty-dir/0c79347d-eb2e-49b2-af27-48ddcf86b7bb-home\") pod \"0c79347d-eb2e-49b2-af27-48ddcf86b7bb\" (UID: \"0c79347d-eb2e-49b2-af27-48ddcf86b7bb\") "
Nov 01 05:59:22.995397 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:22.995293    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca-dshm\" (UniqueName: \"kubernetes.io/empty-dir/05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca-dshm\") pod \"05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca\" (UID: \"05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca\") "
Nov 01 05:59:22.995397 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:22.995349    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c79347d-eb2e-49b2-af27-48ddcf86b7bb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c79347d-eb2e-49b2-af27-48ddcf86b7bb-dshm\") pod \"0c79347d-eb2e-49b2-af27-48ddcf86b7bb\" (UID: \"0c79347d-eb2e-49b2-af27-48ddcf86b7bb\") "
Nov 01 05:59:22.995397 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:22.995371    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca-model-cache" pod "05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca" (UID: "05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:59:22.995397 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:22.995386    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c79347d-eb2e-49b2-af27-48ddcf86b7bb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c79347d-eb2e-49b2-af27-48ddcf86b7bb-tmp-dir\") pod \"0c79347d-eb2e-49b2-af27-48ddcf86b7bb\" (UID: \"0c79347d-eb2e-49b2-af27-48ddcf86b7bb\") "
Nov 01 05:59:22.995666 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:22.995507    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca-kube-api-access-hqkd6\" (UniqueName: \"kubernetes.io/projected/05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca-kube-api-access-hqkd6\") pod \"05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca\" (UID: \"05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca\") "
Nov 01 05:59:22.995666 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:22.995526    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca-home" pod "05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca" (UID: "05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:59:22.995666 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:22.995533    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c79347d-eb2e-49b2-af27-48ddcf86b7bb-home" pod "0c79347d-eb2e-49b2-af27-48ddcf86b7bb" (UID: "0c79347d-eb2e-49b2-af27-48ddcf86b7bb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:59:22.995666 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:22.995555    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca-tmp-dir\") pod \"05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca\" (UID: \"05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca\") "
Nov 01 05:59:22.995666 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:22.995587    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c79347d-eb2e-49b2-af27-48ddcf86b7bb-tls-certs\" (UniqueName: \"kubernetes.io/secret/0c79347d-eb2e-49b2-af27-48ddcf86b7bb-tls-certs\") pod \"0c79347d-eb2e-49b2-af27-48ddcf86b7bb\" (UID: \"0c79347d-eb2e-49b2-af27-48ddcf86b7bb\") "
Nov 01 05:59:22.995666 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:22.995614    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c79347d-eb2e-49b2-af27-48ddcf86b7bb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c79347d-eb2e-49b2-af27-48ddcf86b7bb-model-cache\") pod \"0c79347d-eb2e-49b2-af27-48ddcf86b7bb\" (UID: \"0c79347d-eb2e-49b2-af27-48ddcf86b7bb\") "
Nov 01 05:59:22.995666 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:22.995623    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c79347d-eb2e-49b2-af27-48ddcf86b7bb-tmp-dir" pod "0c79347d-eb2e-49b2-af27-48ddcf86b7bb" (UID: "0c79347d-eb2e-49b2-af27-48ddcf86b7bb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:59:22.996198 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:22.995887    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca-tmp-dir" pod "05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca" (UID: "05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:59:22.996198 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:22.996067    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c79347d-eb2e-49b2-af27-48ddcf86b7bb-model-cache" pod "0c79347d-eb2e-49b2-af27-48ddcf86b7bb" (UID: "0c79347d-eb2e-49b2-af27-48ddcf86b7bb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:59:22.996198 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:22.996152    5689 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c79347d-eb2e-49b2-af27-48ddcf86b7bb-tmp-dir\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:59:22.996198 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:22.996166    5689 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca-tmp-dir\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:59:22.996198 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:22.996174    5689 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca-model-cache\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:59:22.996198 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:22.996185    5689 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca-home\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:59:22.996198 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:22.996199    5689 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c79347d-eb2e-49b2-af27-48ddcf86b7bb-home\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:59:22.997767 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:22.997745    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca-tls-certs" pod "05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca" (UID: "05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 05:59:22.997940 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:22.997929    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c79347d-eb2e-49b2-af27-48ddcf86b7bb-kube-api-access-5vrbr" pod "0c79347d-eb2e-49b2-af27-48ddcf86b7bb" (UID: "0c79347d-eb2e-49b2-af27-48ddcf86b7bb"). InnerVolumeSpecName "kube-api-access-5vrbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:59:22.998205 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:22.998185    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c79347d-eb2e-49b2-af27-48ddcf86b7bb-dshm" pod "0c79347d-eb2e-49b2-af27-48ddcf86b7bb" (UID: "0c79347d-eb2e-49b2-af27-48ddcf86b7bb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:59:22.998277 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:22.998247    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c79347d-eb2e-49b2-af27-48ddcf86b7bb-tls-certs" pod "0c79347d-eb2e-49b2-af27-48ddcf86b7bb" (UID: "0c79347d-eb2e-49b2-af27-48ddcf86b7bb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 05:59:22.998427 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:22.998405    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca-kube-api-access-hqkd6" pod "05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca" (UID: "05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca"). InnerVolumeSpecName "kube-api-access-hqkd6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:59:22.998602 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:22.998587    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca-dshm" pod "05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca" (UID: "05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:59:23.097383 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:23.097346    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vrbr\" (UniqueName: \"kubernetes.io/projected/0c79347d-eb2e-49b2-af27-48ddcf86b7bb-kube-api-access-5vrbr\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:59:23.097383 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:23.097382    5689 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca-tls-certs\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:59:23.097609 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:23.097398    5689 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca-dshm\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:59:23.097609 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:23.097412    5689 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c79347d-eb2e-49b2-af27-48ddcf86b7bb-dshm\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:59:23.097609 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:23.097426    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqkd6\" (UniqueName: \"kubernetes.io/projected/05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca-kube-api-access-hqkd6\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:59:23.097609 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:23.097447    5689 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c79347d-eb2e-49b2-af27-48ddcf86b7bb-tls-certs\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:59:23.097609 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:23.097480    5689 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c79347d-eb2e-49b2-af27-48ddcf86b7bb-model-cache\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:59:23.255882 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:23.255843    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:23.424683 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:23.424648    5689 generic.go:356] "Generic (PLEG): container finished" podID="05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca" containerID="eb663254760978ac1c5ae561a4582a7bfe3186f5a4e0e410679c39fb34422a64" exitCode=0
Nov 01 05:59:23.424860 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:23.424705    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-3d70ff/simulator-w2-3d70ff-kserve-898c5fb4f-t5bzk"
Nov 01 05:59:23.424860 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:23.424731    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-3d70ff/simulator-w2-3d70ff-kserve-898c5fb4f-t5bzk" event={"ID":"05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca","Type":"ContainerDied","Data":"eb663254760978ac1c5ae561a4582a7bfe3186f5a4e0e410679c39fb34422a64"}
Nov 01 05:59:23.424860 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:23.424777    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-3d70ff/simulator-w2-3d70ff-kserve-898c5fb4f-t5bzk" event={"ID":"05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca","Type":"ContainerDied","Data":"67458bc19b71123ee29836a96acf684108e2f6f9a442250d2612018d1a9f6fc8"}
Nov 01 05:59:23.424860 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:23.424800    5689 scope.go:122] "RemoveContainer" containerID="eb663254760978ac1c5ae561a4582a7bfe3186f5a4e0e410679c39fb34422a64"
Nov 01 05:59:23.426299 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:23.426277    5689 generic.go:356] "Generic (PLEG): container finished" podID="0c79347d-eb2e-49b2-af27-48ddcf86b7bb" containerID="43ab0c1f2c3c4e1ea9d4f3b52217df17a7c8d5d9aab7c8b3959f70b2c440c412" exitCode=0
Nov 01 05:59:23.426632 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:23.426322    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-3d70ff/distinct-2-w2-3d70ff-kserve-868b775cd4-nhx4f" event={"ID":"0c79347d-eb2e-49b2-af27-48ddcf86b7bb","Type":"ContainerDied","Data":"43ab0c1f2c3c4e1ea9d4f3b52217df17a7c8d5d9aab7c8b3959f70b2c440c412"}
Nov 01 05:59:23.426632 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:23.426339    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-3d70ff/distinct-2-w2-3d70ff-kserve-868b775cd4-nhx4f"
Nov 01 05:59:23.426632 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:23.426346    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-3d70ff/distinct-2-w2-3d70ff-kserve-868b775cd4-nhx4f" event={"ID":"0c79347d-eb2e-49b2-af27-48ddcf86b7bb","Type":"ContainerDied","Data":"90fb9e6596293c6db025197add3e4485e6b17ee8953739f608390cbef3c50339"}
Nov 01 05:59:23.435416 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:23.435400    5689 scope.go:122] "RemoveContainer" containerID="eb663254760978ac1c5ae561a4582a7bfe3186f5a4e0e410679c39fb34422a64"
Nov 01 05:59:23.435684 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:59:23.435658    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb663254760978ac1c5ae561a4582a7bfe3186f5a4e0e410679c39fb34422a64\": container with ID starting with eb663254760978ac1c5ae561a4582a7bfe3186f5a4e0e410679c39fb34422a64 not found: ID does not exist" containerID="eb663254760978ac1c5ae561a4582a7bfe3186f5a4e0e410679c39fb34422a64"
Nov 01 05:59:23.435759 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:23.435692    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb663254760978ac1c5ae561a4582a7bfe3186f5a4e0e410679c39fb34422a64"} err="failed to get container status \"eb663254760978ac1c5ae561a4582a7bfe3186f5a4e0e410679c39fb34422a64\": rpc error: code = NotFound desc = could not find container \"eb663254760978ac1c5ae561a4582a7bfe3186f5a4e0e410679c39fb34422a64\": container with ID starting with eb663254760978ac1c5ae561a4582a7bfe3186f5a4e0e410679c39fb34422a64 not found: ID does not exist"
Nov 01 05:59:23.435759 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:23.435708    5689 scope.go:122] "RemoveContainer" containerID="43ab0c1f2c3c4e1ea9d4f3b52217df17a7c8d5d9aab7c8b3959f70b2c440c412"
Nov 01 05:59:23.444528 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:23.444502    5689 scope.go:122] "RemoveContainer" containerID="43ab0c1f2c3c4e1ea9d4f3b52217df17a7c8d5d9aab7c8b3959f70b2c440c412"
Nov 01 05:59:23.444789 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:59:23.444760    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43ab0c1f2c3c4e1ea9d4f3b52217df17a7c8d5d9aab7c8b3959f70b2c440c412\": container with ID starting with 43ab0c1f2c3c4e1ea9d4f3b52217df17a7c8d5d9aab7c8b3959f70b2c440c412 not found: ID does not exist" containerID="43ab0c1f2c3c4e1ea9d4f3b52217df17a7c8d5d9aab7c8b3959f70b2c440c412"
Nov 01 05:59:23.444882 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:23.444793    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43ab0c1f2c3c4e1ea9d4f3b52217df17a7c8d5d9aab7c8b3959f70b2c440c412"} err="failed to get container status \"43ab0c1f2c3c4e1ea9d4f3b52217df17a7c8d5d9aab7c8b3959f70b2c440c412\": rpc error: code = NotFound desc = could not find container \"43ab0c1f2c3c4e1ea9d4f3b52217df17a7c8d5d9aab7c8b3959f70b2c440c412\": container with ID starting with 43ab0c1f2c3c4e1ea9d4f3b52217df17a7c8d5d9aab7c8b3959f70b2c440c412 not found: ID does not exist"
Nov 01 05:59:23.587143 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:23.587075    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-3d70ff/distinct-2-w2-3d70ff-kserve-868b775cd4-nhx4f"]
Nov 01 05:59:23.592784 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:23.592756    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-3d70ff/distinct-2-w2-3d70ff-kserve-868b775cd4-nhx4f"]
Nov 01 05:59:23.700603 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:23.700577    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-3d70ff/simulator-w2-3d70ff-kserve-898c5fb4f-t5bzk"]
Nov 01 05:59:23.709845 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:23.709816    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-3d70ff/simulator-w2-3d70ff-kserve-898c5fb4f-t5bzk"]
Nov 01 05:59:24.068084 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:24.068052    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca" path="/var/lib/kubelet/pods/05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca/volumes"
Nov 01 05:59:24.068518 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:24.068504    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c79347d-eb2e-49b2-af27-48ddcf86b7bb" path="/var/lib/kubelet/pods/0c79347d-eb2e-49b2-af27-48ddcf86b7bb/volumes"
Nov 01 05:59:25.605686 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:25.605654    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:25.794567 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:25.794531    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:25.976311 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:25.976283    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:26.172662 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:26.172609    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:26.357938 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:26.357861    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:26.372749 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:26.372723    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-c7f95pfdk" podUID="eaed5d67-d647-4f94-8f6f-446d14c2d2a1" containerName="payload-processing" containerID="cri-o://5950624539e7a47bcbad93dc44fc0542e41b0ac032a3d7f05e2fdcbb590a040b" gracePeriod=25
Nov 01 05:59:26.640106 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:26.640081    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-c7f95pfdk"
Nov 01 05:59:26.744478 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:26.738982    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eaed5d67-d647-4f94-8f6f-446d14c2d2a1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaed5d67-d647-4f94-8f6f-446d14c2d2a1-plugins-config-volume\") pod \"eaed5d67-d647-4f94-8f6f-446d14c2d2a1\" (UID: \"eaed5d67-d647-4f94-8f6f-446d14c2d2a1\") "
Nov 01 05:59:26.744478 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:26.739034    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaed5d67-d647-4f94-8f6f-446d14c2d2a1-provider-credentials\" (UniqueName: \"kubernetes.io/projected/eaed5d67-d647-4f94-8f6f-446d14c2d2a1-provider-credentials\") pod \"eaed5d67-d647-4f94-8f6f-446d14c2d2a1\" (UID: \"eaed5d67-d647-4f94-8f6f-446d14c2d2a1\") "
Nov 01 05:59:26.744478 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:26.739214    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eaed5d67-d647-4f94-8f6f-446d14c2d2a1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/eaed5d67-d647-4f94-8f6f-446d14c2d2a1-routing-overlay-volume\") pod \"eaed5d67-d647-4f94-8f6f-446d14c2d2a1\" (UID: \"eaed5d67-d647-4f94-8f6f-446d14c2d2a1\") "
Nov 01 05:59:26.744478 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:26.739266    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaed5d67-d647-4f94-8f6f-446d14c2d2a1-kube-api-access-mrl9k\" (UniqueName: \"kubernetes.io/projected/eaed5d67-d647-4f94-8f6f-446d14c2d2a1-kube-api-access-mrl9k\") pod \"eaed5d67-d647-4f94-8f6f-446d14c2d2a1\" (UID: \"eaed5d67-d647-4f94-8f6f-446d14c2d2a1\") "
Nov 01 05:59:26.744478 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:26.740181    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaed5d67-d647-4f94-8f6f-446d14c2d2a1-plugins-config-volume" pod "eaed5d67-d647-4f94-8f6f-446d14c2d2a1" (UID: "eaed5d67-d647-4f94-8f6f-446d14c2d2a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:59:26.744478 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:26.742807    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaed5d67-d647-4f94-8f6f-446d14c2d2a1-routing-overlay-volume" pod "eaed5d67-d647-4f94-8f6f-446d14c2d2a1" (UID: "eaed5d67-d647-4f94-8f6f-446d14c2d2a1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:59:26.745627 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:26.745585    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaed5d67-d647-4f94-8f6f-446d14c2d2a1-provider-credentials" pod "eaed5d67-d647-4f94-8f6f-446d14c2d2a1" (UID: "eaed5d67-d647-4f94-8f6f-446d14c2d2a1"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:59:26.771487 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:26.753936    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaed5d67-d647-4f94-8f6f-446d14c2d2a1-kube-api-access-mrl9k" pod "eaed5d67-d647-4f94-8f6f-446d14c2d2a1" (UID: "eaed5d67-d647-4f94-8f6f-446d14c2d2a1"). InnerVolumeSpecName "kube-api-access-mrl9k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:59:26.868563 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:26.852685    5689 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaed5d67-d647-4f94-8f6f-446d14c2d2a1-plugins-config-volume\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:59:26.868563 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:26.852722    5689 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/eaed5d67-d647-4f94-8f6f-446d14c2d2a1-provider-credentials\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:59:26.868563 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:26.852735    5689 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/eaed5d67-d647-4f94-8f6f-446d14c2d2a1-routing-overlay-volume\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:59:26.868563 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:26.852747    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrl9k\" (UniqueName: \"kubernetes.io/projected/eaed5d67-d647-4f94-8f6f-446d14c2d2a1-kube-api-access-mrl9k\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:59:27.454553 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:27.454516    5689 generic.go:356] "Generic (PLEG): container finished" podID="eaed5d67-d647-4f94-8f6f-446d14c2d2a1" containerID="5950624539e7a47bcbad93dc44fc0542e41b0ac032a3d7f05e2fdcbb590a040b" exitCode=0
Nov 01 05:59:27.454732 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:27.454580    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-c7f95pfdk"
Nov 01 05:59:27.454732 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:27.454600    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-c7f95pfdk" event={"ID":"eaed5d67-d647-4f94-8f6f-446d14c2d2a1","Type":"ContainerDied","Data":"5950624539e7a47bcbad93dc44fc0542e41b0ac032a3d7f05e2fdcbb590a040b"}
Nov 01 05:59:27.454732 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:27.454679    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-c7f95pfdk" event={"ID":"eaed5d67-d647-4f94-8f6f-446d14c2d2a1","Type":"ContainerDied","Data":"389ff23d5e98be1e117fe90634bab4a8764508b679e7ff0a5d57231f53b7639b"}
Nov 01 05:59:27.454732 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:27.454707    5689 scope.go:122] "RemoveContainer" containerID="5950624539e7a47bcbad93dc44fc0542e41b0ac032a3d7f05e2fdcbb590a040b"
Nov 01 05:59:27.465305 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:27.465285    5689 scope.go:122] "RemoveContainer" containerID="5950624539e7a47bcbad93dc44fc0542e41b0ac032a3d7f05e2fdcbb590a040b"
Nov 01 05:59:27.465581 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:59:27.465559    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5950624539e7a47bcbad93dc44fc0542e41b0ac032a3d7f05e2fdcbb590a040b\": container with ID starting with 5950624539e7a47bcbad93dc44fc0542e41b0ac032a3d7f05e2fdcbb590a040b not found: ID does not exist" containerID="5950624539e7a47bcbad93dc44fc0542e41b0ac032a3d7f05e2fdcbb590a040b"
Nov 01 05:59:27.465638 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:27.465591    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5950624539e7a47bcbad93dc44fc0542e41b0ac032a3d7f05e2fdcbb590a040b"} err="failed to get container status \"5950624539e7a47bcbad93dc44fc0542e41b0ac032a3d7f05e2fdcbb590a040b\": rpc error: code = NotFound desc = could not find container \"5950624539e7a47bcbad93dc44fc0542e41b0ac032a3d7f05e2fdcbb590a040b\": container with ID starting with 5950624539e7a47bcbad93dc44fc0542e41b0ac032a3d7f05e2fdcbb590a040b not found: ID does not exist"
Nov 01 05:59:27.578115 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:27.578071    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:27.621756 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:27.621725    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-c7f95pfdk"]
Nov 01 05:59:27.625329 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:27.625308    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-c7f95pfdk"]
Nov 01 05:59:28.010676 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:28.010649    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:28.068248 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:28.068219    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eaed5d67-d647-4f94-8f6f-446d14c2d2a1" path="/var/lib/kubelet/pods/eaed5d67-d647-4f94-8f6f-446d14c2d2a1/volumes"
Nov 01 05:59:28.544857 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:28.544828    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:29.289099 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:29.289055    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:29.467851 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:29.467809    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:30.023897 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:30.023828    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:30.110594 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:59:30.110124    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 05:59:30.201357 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:30.201306    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:31.155194 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:31.155159    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-7576pxx5w"]
Nov 01 05:59:31.155770 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:31.155755    5689 cpu_manager.go:414] "containerMap: removing container" podUID="eaed5d67-d647-4f94-8f6f-446d14c2d2a1" containerName="payload-processing"
Nov 01 05:59:31.155826 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:31.155772    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaed5d67-d647-4f94-8f6f-446d14c2d2a1" containerName="payload-processing"
Nov 01 05:59:31.155826 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:31.155816    5689 cpu_manager.go:414] "containerMap: removing container" podUID="0c79347d-eb2e-49b2-af27-48ddcf86b7bb" containerName="main"
Nov 01 05:59:31.155826 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:31.155822    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c79347d-eb2e-49b2-af27-48ddcf86b7bb" containerName="main"
Nov 01 05:59:31.155911 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:31.155831    5689 cpu_manager.go:414] "containerMap: removing container" podUID="05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca" containerName="main"
Nov 01 05:59:31.155911 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:31.155836    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca" containerName="main"
Nov 01 05:59:31.155974 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:31.155924    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c79347d-eb2e-49b2-af27-48ddcf86b7bb" containerName="main"
Nov 01 05:59:31.155974 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:31.155932    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="05b70f5f-3a0f-439b-9f0b-a1e9e76ab4ca" containerName="main"
Nov 01 05:59:31.155974 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:31.155940    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaed5d67-d647-4f94-8f6f-446d14c2d2a1" containerName="payload-processing"
Nov 01 05:59:31.160591 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:31.160573    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-7576pxx5w"
Nov 01 05:59:31.215088 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:31.215067    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-dee6aa\"/\"agc-cred-dc7d983de1b3\""
Nov 01 05:59:31.265043 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:31.265010    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-dee6aa\"/\"routing-overlay\""
Nov 01 05:59:31.265209 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:31.265049    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-dee6aa\"/\"openshift-service-ca.crt\""
Nov 01 05:59:31.265209 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:31.265128    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-dee6aa\"/\"payload-processing-external-model-e2e--19b5d07b-dockercfg-jwthr\""
Nov 01 05:59:31.265323 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:31.265308    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-dee6aa\"/\"kube-root-ca.crt\""
Nov 01 05:59:31.265381 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:31.265367    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-dee6aa\"/\"payload-processing-external-model-plugins-e2e-extmod-a-dee6aa\""
Nov 01 05:59:31.267660 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:31.267640    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-7576pxx5w"]
Nov 01 05:59:31.333545 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:31.333515    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2c9b958-8b54-4e3e-abd1-e755d7b1b829-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-7576pxx5w\" (UID: \"a2c9b958-8b54-4e3e-abd1-e755d7b1b829\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-7576pxx5w"
Nov 01 05:59:31.333674 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:31.333628    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a2c9b958-8b54-4e3e-abd1-e755d7b1b829-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-7576pxx5w\" (UID: \"a2c9b958-8b54-4e3e-abd1-e755d7b1b829\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-7576pxx5w"
Nov 01 05:59:31.333778 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:31.333757    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnvdv\" (UniqueName: \"kubernetes.io/projected/a2c9b958-8b54-4e3e-abd1-e755d7b1b829-kube-api-access-dnvdv\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-7576pxx5w\" (UID: \"a2c9b958-8b54-4e3e-abd1-e755d7b1b829\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-7576pxx5w"
Nov 01 05:59:31.333882 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:31.333865    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a2c9b958-8b54-4e3e-abd1-e755d7b1b829-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-7576pxx5w\" (UID: \"a2c9b958-8b54-4e3e-abd1-e755d7b1b829\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-7576pxx5w"
Nov 01 05:59:31.435285 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:31.435214    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnvdv\" (UniqueName: \"kubernetes.io/projected/a2c9b958-8b54-4e3e-abd1-e755d7b1b829-kube-api-access-dnvdv\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-7576pxx5w\" (UID: \"a2c9b958-8b54-4e3e-abd1-e755d7b1b829\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-7576pxx5w"
Nov 01 05:59:31.435285 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:31.435281    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a2c9b958-8b54-4e3e-abd1-e755d7b1b829-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-7576pxx5w\" (UID: \"a2c9b958-8b54-4e3e-abd1-e755d7b1b829\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-7576pxx5w"
Nov 01 05:59:31.435499 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:31.435319    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2c9b958-8b54-4e3e-abd1-e755d7b1b829-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-7576pxx5w\" (UID: \"a2c9b958-8b54-4e3e-abd1-e755d7b1b829\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-7576pxx5w"
Nov 01 05:59:31.435499 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:31.435355    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a2c9b958-8b54-4e3e-abd1-e755d7b1b829-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-7576pxx5w\" (UID: \"a2c9b958-8b54-4e3e-abd1-e755d7b1b829\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-7576pxx5w"
Nov 01 05:59:31.435499 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:59:31.435416    5689 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-dee6aa/agc-cred-dc7d983de1b3: secret "agc-cred-dc7d983de1b3" not found
Nov 01 05:59:31.435499 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:59:31.435434    5689 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-7576pxx5w: secret "agc-cred-dc7d983de1b3" not found
Nov 01 05:59:31.435730 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:59:31.435523    5689 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a2c9b958-8b54-4e3e-abd1-e755d7b1b829-provider-credentials podName:a2c9b958-8b54-4e3e-abd1-e755d7b1b829 nodeName:}" failed. No retries permitted until 2026-11-01 05:59:31.93550232 +0000 UTC m=+3350.006299020 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/a2c9b958-8b54-4e3e-abd1-e755d7b1b829-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-dee6aa-7576pxx5w" (UID: "a2c9b958-8b54-4e3e-abd1-e755d7b1b829") : secret "agc-cred-dc7d983de1b3" not found
Nov 01 05:59:31.435993 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:31.435974    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a2c9b958-8b54-4e3e-abd1-e755d7b1b829-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-7576pxx5w\" (UID: \"a2c9b958-8b54-4e3e-abd1-e755d7b1b829\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-7576pxx5w"
Nov 01 05:59:31.436049 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:31.436030    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2c9b958-8b54-4e3e-abd1-e755d7b1b829-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-7576pxx5w\" (UID: \"a2c9b958-8b54-4e3e-abd1-e755d7b1b829\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-7576pxx5w"
Nov 01 05:59:31.496221 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:31.496197    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnvdv\" (UniqueName: \"kubernetes.io/projected/a2c9b958-8b54-4e3e-abd1-e755d7b1b829-kube-api-access-dnvdv\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-7576pxx5w\" (UID: \"a2c9b958-8b54-4e3e-abd1-e755d7b1b829\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-7576pxx5w"
Nov 01 05:59:31.877769 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:31.877690    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 05:59:31.941034 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:31.940999    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a2c9b958-8b54-4e3e-abd1-e755d7b1b829-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-7576pxx5w\" (UID: \"a2c9b958-8b54-4e3e-abd1-e755d7b1b829\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-7576pxx5w"
Nov 01 05:59:31.941192 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:59:31.941153    5689 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-dee6aa/agc-cred-dc7d983de1b3: secret "agc-cred-dc7d983de1b3" not found
Nov 01 05:59:31.941192 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:59:31.941170    5689 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-7576pxx5w: secret "agc-cred-dc7d983de1b3" not found
Nov 01 05:59:31.941267 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:59:31.941235    5689 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a2c9b958-8b54-4e3e-abd1-e755d7b1b829-provider-credentials podName:a2c9b958-8b54-4e3e-abd1-e755d7b1b829 nodeName:}" failed. No retries permitted until 2026-11-01 05:59:32.94121912 +0000 UTC m=+3351.012015820 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/a2c9b958-8b54-4e3e-abd1-e755d7b1b829-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-dee6aa-7576pxx5w" (UID: "a2c9b958-8b54-4e3e-abd1-e755d7b1b829") : secret "agc-cred-dc7d983de1b3" not found
Nov 01 05:59:32.757468 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:32.757419    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:32.963741 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:32.963085    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a2c9b958-8b54-4e3e-abd1-e755d7b1b829-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-7576pxx5w\" (UID: \"a2c9b958-8b54-4e3e-abd1-e755d7b1b829\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-7576pxx5w"
Nov 01 05:59:32.964026 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:59:32.963250    5689 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-dee6aa/agc-cred-dc7d983de1b3: secret "agc-cred-dc7d983de1b3" not found
Nov 01 05:59:32.964130 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:59:32.964118    5689 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-7576pxx5w: secret "agc-cred-dc7d983de1b3" not found
Nov 01 05:59:32.964243 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:59:32.964234    5689 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a2c9b958-8b54-4e3e-abd1-e755d7b1b829-provider-credentials podName:a2c9b958-8b54-4e3e-abd1-e755d7b1b829 nodeName:}" failed. No retries permitted until 2026-11-01 05:59:34.964214469 +0000 UTC m=+3353.035011172 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/a2c9b958-8b54-4e3e-abd1-e755d7b1b829-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-dee6aa-7576pxx5w" (UID: "a2c9b958-8b54-4e3e-abd1-e755d7b1b829") : secret "agc-cred-dc7d983de1b3" not found
Nov 01 05:59:32.965190 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:32.965165    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:33.146821 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:33.146735    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-fd2672-7d58df9c59-2glzq"]
Nov 01 05:59:33.151555 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:33.151533    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-fd2672-7d58df9c59-2glzq"
Nov 01 05:59:33.176858 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:33.176824    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dc7a0fa-fa53-4c98-a342-b3f67b3a1824-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-fd2672-7d58df9c59-2glzq\" (UID: \"0dc7a0fa-fa53-4c98-a342-b3f67b3a1824\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-fd2672-7d58df9c59-2glzq"
Nov 01 05:59:33.176999 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:33.176983    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjqmd\" (UniqueName: \"kubernetes.io/projected/0dc7a0fa-fa53-4c98-a342-b3f67b3a1824-kube-api-access-vjqmd\") pod \"payload-pre-processing-e2e-shared-b-w1-fd2672-7d58df9c59-2glzq\" (UID: \"0dc7a0fa-fa53-4c98-a342-b3f67b3a1824\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-fd2672-7d58df9c59-2glzq"
Nov 01 05:59:33.257623 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:33.257582    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-fd2672\""
Nov 01 05:59:33.257623 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:33.257583    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-fd2672-dockercfg-n7z8g\""
Nov 01 05:59:33.290478 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:33.277700    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-fd2672-7d58df9c59-2glzq"]
Nov 01 05:59:33.290478 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:33.278307    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjqmd\" (UniqueName: \"kubernetes.io/projected/0dc7a0fa-fa53-4c98-a342-b3f67b3a1824-kube-api-access-vjqmd\") pod \"payload-pre-processing-e2e-shared-b-w1-fd2672-7d58df9c59-2glzq\" (UID: \"0dc7a0fa-fa53-4c98-a342-b3f67b3a1824\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-fd2672-7d58df9c59-2glzq"
Nov 01 05:59:33.290478 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:33.278489    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dc7a0fa-fa53-4c98-a342-b3f67b3a1824-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-fd2672-7d58df9c59-2glzq\" (UID: \"0dc7a0fa-fa53-4c98-a342-b3f67b3a1824\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-fd2672-7d58df9c59-2glzq"
Nov 01 05:59:33.290478 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:33.279406    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dc7a0fa-fa53-4c98-a342-b3f67b3a1824-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-fd2672-7d58df9c59-2glzq\" (UID: \"0dc7a0fa-fa53-4c98-a342-b3f67b3a1824\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-fd2672-7d58df9c59-2glzq"
Nov 01 05:59:33.323262 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:33.318306    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-fd2672-659f849fc9-pqzgj"]
Nov 01 05:59:33.329024 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:33.328998    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-fd2672-659f849fc9-pqzgj"
Nov 01 05:59:33.366829 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:33.366798    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjqmd\" (UniqueName: \"kubernetes.io/projected/0dc7a0fa-fa53-4c98-a342-b3f67b3a1824-kube-api-access-vjqmd\") pod \"payload-pre-processing-e2e-shared-b-w1-fd2672-7d58df9c59-2glzq\" (UID: \"0dc7a0fa-fa53-4c98-a342-b3f67b3a1824\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-fd2672-7d58df9c59-2glzq"
Nov 01 05:59:33.384832 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:33.384692    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-fd2672-659f849fc9-pqzgj"]
Nov 01 05:59:33.465987 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:33.465900    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-fd2672-7d58df9c59-2glzq"
Nov 01 05:59:33.481943 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:33.481900    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75a0b394-e687-407f-8a86-4f2100a6dfc6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-fd2672-659f849fc9-pqzgj\" (UID: \"75a0b394-e687-407f-8a86-4f2100a6dfc6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-fd2672-659f849fc9-pqzgj"
Nov 01 05:59:33.482244 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:33.482218    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw4pp\" (UniqueName: \"kubernetes.io/projected/75a0b394-e687-407f-8a86-4f2100a6dfc6-kube-api-access-rw4pp\") pod \"payload-processing-e2e-shared-b-w1-fd2672-659f849fc9-pqzgj\" (UID: \"75a0b394-e687-407f-8a86-4f2100a6dfc6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-fd2672-659f849fc9-pqzgj"
Nov 01 05:59:33.583294 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:33.583252    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw4pp\" (UniqueName: \"kubernetes.io/projected/75a0b394-e687-407f-8a86-4f2100a6dfc6-kube-api-access-rw4pp\") pod \"payload-processing-e2e-shared-b-w1-fd2672-659f849fc9-pqzgj\" (UID: \"75a0b394-e687-407f-8a86-4f2100a6dfc6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-fd2672-659f849fc9-pqzgj"
Nov 01 05:59:33.583513 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:33.583493    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75a0b394-e687-407f-8a86-4f2100a6dfc6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-fd2672-659f849fc9-pqzgj\" (UID: \"75a0b394-e687-407f-8a86-4f2100a6dfc6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-fd2672-659f849fc9-pqzgj"
Nov 01 05:59:33.584222 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:33.584201    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75a0b394-e687-407f-8a86-4f2100a6dfc6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-fd2672-659f849fc9-pqzgj\" (UID: \"75a0b394-e687-407f-8a86-4f2100a6dfc6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-fd2672-659f849fc9-pqzgj"
Nov 01 05:59:33.647006 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:33.646971    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw4pp\" (UniqueName: \"kubernetes.io/projected/75a0b394-e687-407f-8a86-4f2100a6dfc6-kube-api-access-rw4pp\") pod \"payload-processing-e2e-shared-b-w1-fd2672-659f849fc9-pqzgj\" (UID: \"75a0b394-e687-407f-8a86-4f2100a6dfc6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-fd2672-659f849fc9-pqzgj"
Nov 01 05:59:33.668412 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:33.668383    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-fd2672-659f849fc9-pqzgj"
Nov 01 05:59:33.855262 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:33.855208    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-fd2672-7d58df9c59-2glzq"]
Nov 01 05:59:33.858836 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:59:33.858809    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dc7a0fa_fa53_4c98_a342_b3f67b3a1824.slice/crio-23ed5dfcc8e092f9bb32aad9fe48fdb93c5ce384b5577c283b0106ea7caa1cec WatchSource:0}: Error finding container 23ed5dfcc8e092f9bb32aad9fe48fdb93c5ce384b5577c283b0106ea7caa1cec: Status 404 returned error can't find the container with id 23ed5dfcc8e092f9bb32aad9fe48fdb93c5ce384b5577c283b0106ea7caa1cec
Nov 01 05:59:33.881983 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:33.881955    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:33.954535 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:33.952303    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-7576pxx5w"]
Nov 01 05:59:33.954535 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:59:33.952761    5689 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-7576pxx5w" podUID="a2c9b958-8b54-4e3e-abd1-e755d7b1b829"
Nov 01 05:59:34.069899 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:34.069837    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-fd2672-659f849fc9-pqzgj"]
Nov 01 05:59:34.072058 ip-10-0-129-92 kubenswrapper[5689]: W1101 05:59:34.072031    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75a0b394_e687_407f_8a86_4f2100a6dfc6.slice/crio-ff7e6339ff7529e872858962d2a931c19b053a752f4e9c6d267496b2ee14173c WatchSource:0}: Error finding container ff7e6339ff7529e872858962d2a931c19b053a752f4e9c6d267496b2ee14173c: Status 404 returned error can't find the container with id ff7e6339ff7529e872858962d2a931c19b053a752f4e9c6d267496b2ee14173c
Nov 01 05:59:34.134037 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:34.134003    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:34.504628 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:34.504592    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-fd2672-659f849fc9-pqzgj" event={"ID":"75a0b394-e687-407f-8a86-4f2100a6dfc6","Type":"ContainerStarted","Data":"5cadde8ea56c19d44f37d4191e0e4be3b02ad73dc04cca43cdddffe095d008d4"}
Nov 01 05:59:34.504822 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:34.504653    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-fd2672-659f849fc9-pqzgj" event={"ID":"75a0b394-e687-407f-8a86-4f2100a6dfc6","Type":"ContainerStarted","Data":"ff7e6339ff7529e872858962d2a931c19b053a752f4e9c6d267496b2ee14173c"}
Nov 01 05:59:34.504822 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:34.504682    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-fd2672-659f849fc9-pqzgj"
Nov 01 05:59:34.506253 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:34.506223    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-fd2672-7d58df9c59-2glzq" event={"ID":"0dc7a0fa-fa53-4c98-a342-b3f67b3a1824","Type":"ContainerStarted","Data":"f8406bc94f62b89ad46b1c26b1ce28a978b1881a7383fd2523ed439390165a06"}
Nov 01 05:59:34.506388 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:34.506268    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-fd2672-7d58df9c59-2glzq" event={"ID":"0dc7a0fa-fa53-4c98-a342-b3f67b3a1824","Type":"ContainerStarted","Data":"23ed5dfcc8e092f9bb32aad9fe48fdb93c5ce384b5577c283b0106ea7caa1cec"}
Nov 01 05:59:34.506388 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:34.506295    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-fd2672-7d58df9c59-2glzq"
Nov 01 05:59:34.506388 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:34.506312    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-7576pxx5w"
Nov 01 05:59:34.513971 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:34.513952    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-7576pxx5w"
Nov 01 05:59:34.632601 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:34.632545    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-fd2672-659f849fc9-pqzgj" podStartSLOduration=1.632530646 podStartE2EDuration="1.632530646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:59:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:59:34.631479086 +0000 UTC m=+3352.702275807" watchObservedRunningTime="2026-11-01 05:59:34.632530646 +0000 UTC m=+3352.703327391"
Nov 01 05:59:34.703235 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:34.703203    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2c9b958-8b54-4e3e-abd1-e755d7b1b829-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2c9b958-8b54-4e3e-abd1-e755d7b1b829-plugins-config-volume\") pod \"a2c9b958-8b54-4e3e-abd1-e755d7b1b829\" (UID: \"a2c9b958-8b54-4e3e-abd1-e755d7b1b829\") "
Nov 01 05:59:34.703410 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:34.703245    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2c9b958-8b54-4e3e-abd1-e755d7b1b829-kube-api-access-dnvdv\" (UniqueName: \"kubernetes.io/projected/a2c9b958-8b54-4e3e-abd1-e755d7b1b829-kube-api-access-dnvdv\") pod \"a2c9b958-8b54-4e3e-abd1-e755d7b1b829\" (UID: \"a2c9b958-8b54-4e3e-abd1-e755d7b1b829\") "
Nov 01 05:59:34.703410 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:34.703286    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2c9b958-8b54-4e3e-abd1-e755d7b1b829-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a2c9b958-8b54-4e3e-abd1-e755d7b1b829-routing-overlay-volume\") pod \"a2c9b958-8b54-4e3e-abd1-e755d7b1b829\" (UID: \"a2c9b958-8b54-4e3e-abd1-e755d7b1b829\") "
Nov 01 05:59:34.703707 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:34.703682    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2c9b958-8b54-4e3e-abd1-e755d7b1b829-plugins-config-volume" pod "a2c9b958-8b54-4e3e-abd1-e755d7b1b829" (UID: "a2c9b958-8b54-4e3e-abd1-e755d7b1b829"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:59:34.703767 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:34.703728    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2c9b958-8b54-4e3e-abd1-e755d7b1b829-routing-overlay-volume" pod "a2c9b958-8b54-4e3e-abd1-e755d7b1b829" (UID: "a2c9b958-8b54-4e3e-abd1-e755d7b1b829"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:59:34.705562 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:34.705539    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2c9b958-8b54-4e3e-abd1-e755d7b1b829-kube-api-access-dnvdv" pod "a2c9b958-8b54-4e3e-abd1-e755d7b1b829" (UID: "a2c9b958-8b54-4e3e-abd1-e755d7b1b829"). InnerVolumeSpecName "kube-api-access-dnvdv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:59:34.747875 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:34.747831    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-fd2672-7d58df9c59-2glzq" podStartSLOduration=1.747818218 podStartE2EDuration="1.747818218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:59:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:59:34.745955601 +0000 UTC m=+3352.816752320" watchObservedRunningTime="2026-11-01 05:59:34.747818218 +0000 UTC m=+3352.818614938"
Nov 01 05:59:34.804875 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:34.804815    5689 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2c9b958-8b54-4e3e-abd1-e755d7b1b829-plugins-config-volume\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:59:34.804875 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:34.804839    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnvdv\" (UniqueName: \"kubernetes.io/projected/a2c9b958-8b54-4e3e-abd1-e755d7b1b829-kube-api-access-dnvdv\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:59:34.804875 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:34.804848    5689 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a2c9b958-8b54-4e3e-abd1-e755d7b1b829-routing-overlay-volume\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:59:35.008284 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:35.008245    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a2c9b958-8b54-4e3e-abd1-e755d7b1b829-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dee6aa-7576pxx5w\" (UID: \"a2c9b958-8b54-4e3e-abd1-e755d7b1b829\") " pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-7576pxx5w"
Nov 01 05:59:35.008794 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:59:35.008403    5689 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-dee6aa/agc-cred-dc7d983de1b3: secret "agc-cred-dc7d983de1b3" not found
Nov 01 05:59:35.008794 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:59:35.008424    5689 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-7576pxx5w: secret "agc-cred-dc7d983de1b3" not found
Nov 01 05:59:35.008794 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:59:35.008530    5689 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a2c9b958-8b54-4e3e-abd1-e755d7b1b829-provider-credentials podName:a2c9b958-8b54-4e3e-abd1-e755d7b1b829 nodeName:}" failed. No retries permitted until 2026-11-01 05:59:39.008509484 +0000 UTC m=+3357.079306182 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/a2c9b958-8b54-4e3e-abd1-e755d7b1b829-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-dee6aa-7576pxx5w" (UID: "a2c9b958-8b54-4e3e-abd1-e755d7b1b829") : secret "agc-cred-dc7d983de1b3" not found
Nov 01 05:59:35.511436 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:35.511406    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-7576pxx5w"
Nov 01 05:59:35.741419 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:35.741387    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-7576pxx5w"]
Nov 01 05:59:35.747149 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:35.747122    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-dee6aa/payload-processing-external-model-e2e-extmod-a-dee6aa-7576pxx5w"]
Nov 01 05:59:35.918108 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:35.918088    5689 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a2c9b958-8b54-4e3e-abd1-e755d7b1b829-provider-credentials\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:59:36.068033 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:36.068005    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2c9b958-8b54-4e3e-abd1-e755d7b1b829" path="/var/lib/kubelet/pods/a2c9b958-8b54-4e3e-abd1-e755d7b1b829/volumes"
Nov 01 05:59:36.522771 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:36.522732    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:36.702519 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:36.702485    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:39.004875 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:39.004836    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:40.756176 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:40.756148    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:40.944927 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:40.944894    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:42.740899 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:42.740843    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:42.914442 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:42.914401    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:43.837508 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:43.835909    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:45.254768 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:45.254740    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:45.437203 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:45.437171    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:45.513985 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:45.513910    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-fd2672-659f849fc9-pqzgj"
Nov 01 05:59:45.513985 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:45.513973    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-fd2672-7d58df9c59-2glzq"
Nov 01 05:59:46.012432 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:46.012402    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:46.459264 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:46.459235    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:47.325883 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:47.325847    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:47.857194 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:47.857165    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-3d70ff-5ddfbc8c59-7xzdv"]
Nov 01 05:59:47.857652 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:47.857369    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3d70ff-5ddfbc8c59-7xzdv" podUID="0d48b428-8bcb-4777-9e1b-03b8eb99ecfd" containerName="maas-api" containerID="cri-o://9faac4731985ad32caf58e7bb4d2f7bf20ee8215d057ce80c9f0bbf557cd0819" gracePeriod=30
Nov 01 05:59:47.997072 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:47.997026    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:48.186160 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:48.186135    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3d70ff-5ddfbc8c59-7xzdv"
Nov 01 05:59:48.276476 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:48.274868    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d48b428-8bcb-4777-9e1b-03b8eb99ecfd-kube-api-access-vmgj2\" (UniqueName: \"kubernetes.io/projected/0d48b428-8bcb-4777-9e1b-03b8eb99ecfd-kube-api-access-vmgj2\") pod \"0d48b428-8bcb-4777-9e1b-03b8eb99ecfd\" (UID: \"0d48b428-8bcb-4777-9e1b-03b8eb99ecfd\") "
Nov 01 05:59:48.276476 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:48.274989    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d48b428-8bcb-4777-9e1b-03b8eb99ecfd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d48b428-8bcb-4777-9e1b-03b8eb99ecfd-openshift-service-ca\") pod \"0d48b428-8bcb-4777-9e1b-03b8eb99ecfd\" (UID: \"0d48b428-8bcb-4777-9e1b-03b8eb99ecfd\") "
Nov 01 05:59:48.276476 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:48.275080    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d48b428-8bcb-4777-9e1b-03b8eb99ecfd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d48b428-8bcb-4777-9e1b-03b8eb99ecfd-metrics-certs\") pod \"0d48b428-8bcb-4777-9e1b-03b8eb99ecfd\" (UID: \"0d48b428-8bcb-4777-9e1b-03b8eb99ecfd\") "
Nov 01 05:59:48.276476 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:48.275137    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d48b428-8bcb-4777-9e1b-03b8eb99ecfd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d48b428-8bcb-4777-9e1b-03b8eb99ecfd-maas-api-tls\") pod \"0d48b428-8bcb-4777-9e1b-03b8eb99ecfd\" (UID: \"0d48b428-8bcb-4777-9e1b-03b8eb99ecfd\") "
Nov 01 05:59:48.291479 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:48.277821    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d48b428-8bcb-4777-9e1b-03b8eb99ecfd-openshift-service-ca" pod "0d48b428-8bcb-4777-9e1b-03b8eb99ecfd" (UID: "0d48b428-8bcb-4777-9e1b-03b8eb99ecfd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:59:48.305476 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:48.292037    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d48b428-8bcb-4777-9e1b-03b8eb99ecfd-kube-api-access-vmgj2" pod "0d48b428-8bcb-4777-9e1b-03b8eb99ecfd" (UID: "0d48b428-8bcb-4777-9e1b-03b8eb99ecfd"). InnerVolumeSpecName "kube-api-access-vmgj2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:59:48.309260 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:48.309221    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d48b428-8bcb-4777-9e1b-03b8eb99ecfd-maas-api-tls" pod "0d48b428-8bcb-4777-9e1b-03b8eb99ecfd" (UID: "0d48b428-8bcb-4777-9e1b-03b8eb99ecfd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 05:59:48.325968 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:48.325933    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d48b428-8bcb-4777-9e1b-03b8eb99ecfd-metrics-certs" pod "0d48b428-8bcb-4777-9e1b-03b8eb99ecfd" (UID: "0d48b428-8bcb-4777-9e1b-03b8eb99ecfd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 05:59:48.377133 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:48.377099    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmgj2\" (UniqueName: \"kubernetes.io/projected/0d48b428-8bcb-4777-9e1b-03b8eb99ecfd-kube-api-access-vmgj2\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:59:48.377133 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:48.377127    5689 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d48b428-8bcb-4777-9e1b-03b8eb99ecfd-openshift-service-ca\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:59:48.377133 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:48.377138    5689 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d48b428-8bcb-4777-9e1b-03b8eb99ecfd-metrics-certs\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:59:48.377368 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:48.377148    5689 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d48b428-8bcb-4777-9e1b-03b8eb99ecfd-maas-api-tls\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:59:48.587194 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:48.587109    5689 generic.go:356] "Generic (PLEG): container finished" podID="0d48b428-8bcb-4777-9e1b-03b8eb99ecfd" containerID="9faac4731985ad32caf58e7bb4d2f7bf20ee8215d057ce80c9f0bbf557cd0819" exitCode=0
Nov 01 05:59:48.587194 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:48.587152    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3d70ff-5ddfbc8c59-7xzdv"
Nov 01 05:59:48.587415 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:48.587196    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3d70ff-5ddfbc8c59-7xzdv" event={"ID":"0d48b428-8bcb-4777-9e1b-03b8eb99ecfd","Type":"ContainerDied","Data":"9faac4731985ad32caf58e7bb4d2f7bf20ee8215d057ce80c9f0bbf557cd0819"}
Nov 01 05:59:48.587415 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:48.587249    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3d70ff-5ddfbc8c59-7xzdv" event={"ID":"0d48b428-8bcb-4777-9e1b-03b8eb99ecfd","Type":"ContainerDied","Data":"5d4ee91c66286af344b406038418f8176d15905ab0e13f3f905748577937a7d7"}
Nov 01 05:59:48.587415 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:48.587273    5689 scope.go:122] "RemoveContainer" containerID="9faac4731985ad32caf58e7bb4d2f7bf20ee8215d057ce80c9f0bbf557cd0819"
Nov 01 05:59:48.597361 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:48.597346    5689 scope.go:122] "RemoveContainer" containerID="9faac4731985ad32caf58e7bb4d2f7bf20ee8215d057ce80c9f0bbf557cd0819"
Nov 01 05:59:48.597660 ip-10-0-129-92 kubenswrapper[5689]: E1101 05:59:48.597640    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9faac4731985ad32caf58e7bb4d2f7bf20ee8215d057ce80c9f0bbf557cd0819\": container with ID starting with 9faac4731985ad32caf58e7bb4d2f7bf20ee8215d057ce80c9f0bbf557cd0819 not found: ID does not exist" containerID="9faac4731985ad32caf58e7bb4d2f7bf20ee8215d057ce80c9f0bbf557cd0819"
Nov 01 05:59:48.597737 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:48.597673    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9faac4731985ad32caf58e7bb4d2f7bf20ee8215d057ce80c9f0bbf557cd0819"} err="failed to get container status \"9faac4731985ad32caf58e7bb4d2f7bf20ee8215d057ce80c9f0bbf557cd0819\": rpc error: code = NotFound desc = could not find container \"9faac4731985ad32caf58e7bb4d2f7bf20ee8215d057ce80c9f0bbf557cd0819\": container with ID starting with 9faac4731985ad32caf58e7bb4d2f7bf20ee8215d057ce80c9f0bbf557cd0819 not found: ID does not exist"
Nov 01 05:59:48.750618 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:48.750586    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-3d70ff-5ddfbc8c59-7xzdv"]
Nov 01 05:59:48.754732 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:48.754703    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-3d70ff-5ddfbc8c59-7xzdv"]
Nov 01 05:59:49.702701 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:49.702667    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:50.068432 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:50.068360    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d48b428-8bcb-4777-9e1b-03b8eb99ecfd" path="/var/lib/kubelet/pods/0d48b428-8bcb-4777-9e1b-03b8eb99ecfd/volumes"
Nov 01 05:59:51.849717 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:51.849683    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:54.498867 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:54.498837    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:54.686311 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:54.686277    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:54.898434 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:54.898406    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:56.254659 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:56.254628    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:56.704694 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:56.704661    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg"]
Nov 01 05:59:56.704976 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:56.704945    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg" podUID="ba32daa8-34ac-44b9-a090-d4804cdeed41" containerName="istio-proxy" containerID="cri-o://4505d6a083ef4eea3dadeb0a38fdafc33b63545dd3bc826114ca10b7d9af344d" gracePeriod=30
Nov 01 05:59:56.739274 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:56.739246    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:57.991541 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:57.991481    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:58.172482 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:58.172439    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:59.663734 ip-10-0-129-92 kubenswrapper[5689]: I1101 05:59:59.663706    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:00:00.068864 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:00:00.068782    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 06:00:00.192624 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:00.192589    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a48099-29891880-r4lkn"]
Nov 01 06:00:00.193218 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:00.193200    5689 cpu_manager.go:414] "containerMap: removing container" podUID="0d48b428-8bcb-4777-9e1b-03b8eb99ecfd" containerName="maas-api"
Nov 01 06:00:00.193310 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:00.193220    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d48b428-8bcb-4777-9e1b-03b8eb99ecfd" containerName="maas-api"
Nov 01 06:00:00.193310 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:00.193301    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d48b428-8bcb-4777-9e1b-03b8eb99ecfd" containerName="maas-api"
Nov 01 06:00:00.198167 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:00.198149    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a48099-29891880-r4lkn"
Nov 01 06:00:00.213593 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:00.211747    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-897c87-29891880-49f7n"]
Nov 01 06:00:00.216025 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:00.216000    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891880-dkwgt"]
Nov 01 06:00:00.216136 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:00.216103    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-897c87-29891880-49f7n"
Nov 01 06:00:00.219715 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:00.219697    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891880-dkwgt"
Nov 01 06:00:00.328514 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:00.328405    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69fb0a82-951c-43eb-9005-66798d4712b3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-897c87-29891880-49f7n\" (UID: \"69fb0a82-951c-43eb-9005-66798d4712b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-897c87-29891880-49f7n"
Nov 01 06:00:00.328514 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:00.328447    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6chp\" (UniqueName: \"kubernetes.io/projected/69fb0a82-951c-43eb-9005-66798d4712b3-kube-api-access-r6chp\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-897c87-29891880-49f7n\" (UID: \"69fb0a82-951c-43eb-9005-66798d4712b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-897c87-29891880-49f7n"
Nov 01 06:00:00.328741 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:00.328587    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r25p2\" (UniqueName: \"kubernetes.io/projected/bd5575b5-1e5c-4224-8cc0-6a91d0445ddc-kube-api-access-r25p2\") pod \"maas-api-key-cleanup-29891880-dkwgt\" (UID: \"bd5575b5-1e5c-4224-8cc0-6a91d0445ddc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891880-dkwgt"
Nov 01 06:00:00.328741 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:00.328639    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a247d001-e5aa-44f0-9213-b76f6e02dc64-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-a48099-29891880-r4lkn\" (UID: \"a247d001-e5aa-44f0-9213-b76f6e02dc64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a48099-29891880-r4lkn"
Nov 01 06:00:00.328859 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:00.328774    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd5575b5-1e5c-4224-8cc0-6a91d0445ddc-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891880-dkwgt\" (UID: \"bd5575b5-1e5c-4224-8cc0-6a91d0445ddc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891880-dkwgt"
Nov 01 06:00:00.328859 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:00.328804    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkmmb\" (UniqueName: \"kubernetes.io/projected/a247d001-e5aa-44f0-9213-b76f6e02dc64-kube-api-access-pkmmb\") pod \"maas-api-key-cleanup-e2e-worker-w4-a48099-29891880-r4lkn\" (UID: \"a247d001-e5aa-44f0-9213-b76f6e02dc64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a48099-29891880-r4lkn"
Nov 01 06:00:00.339445 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:00.339419    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sp8g2\""
Nov 01 06:00:00.368331 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:00.368303    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-897c87-29891880-49f7n"]
Nov 01 06:00:00.375127 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:00.375100    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891880-dkwgt"]
Nov 01 06:00:00.395426 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:00.393852    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a48099-29891880-r4lkn"]
Nov 01 06:00:00.430162 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:00.430126    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69fb0a82-951c-43eb-9005-66798d4712b3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-897c87-29891880-49f7n\" (UID: \"69fb0a82-951c-43eb-9005-66798d4712b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-897c87-29891880-49f7n"
Nov 01 06:00:00.430360 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:00.430186    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6chp\" (UniqueName: \"kubernetes.io/projected/69fb0a82-951c-43eb-9005-66798d4712b3-kube-api-access-r6chp\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-897c87-29891880-49f7n\" (UID: \"69fb0a82-951c-43eb-9005-66798d4712b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-897c87-29891880-49f7n"
Nov 01 06:00:00.430528 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:00.430508    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r25p2\" (UniqueName: \"kubernetes.io/projected/bd5575b5-1e5c-4224-8cc0-6a91d0445ddc-kube-api-access-r25p2\") pod \"maas-api-key-cleanup-29891880-dkwgt\" (UID: \"bd5575b5-1e5c-4224-8cc0-6a91d0445ddc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891880-dkwgt"
Nov 01 06:00:00.430610 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:00.430560    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a247d001-e5aa-44f0-9213-b76f6e02dc64-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-a48099-29891880-r4lkn\" (UID: \"a247d001-e5aa-44f0-9213-b76f6e02dc64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a48099-29891880-r4lkn"
Nov 01 06:00:00.430658 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:00.430649    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd5575b5-1e5c-4224-8cc0-6a91d0445ddc-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891880-dkwgt\" (UID: \"bd5575b5-1e5c-4224-8cc0-6a91d0445ddc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891880-dkwgt"
Nov 01 06:00:00.430706 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:00.430674    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkmmb\" (UniqueName: \"kubernetes.io/projected/a247d001-e5aa-44f0-9213-b76f6e02dc64-kube-api-access-pkmmb\") pod \"maas-api-key-cleanup-e2e-worker-w4-a48099-29891880-r4lkn\" (UID: \"a247d001-e5aa-44f0-9213-b76f6e02dc64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a48099-29891880-r4lkn"
Nov 01 06:00:00.430986 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:00.430964    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69fb0a82-951c-43eb-9005-66798d4712b3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-897c87-29891880-49f7n\" (UID: \"69fb0a82-951c-43eb-9005-66798d4712b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-897c87-29891880-49f7n"
Nov 01 06:00:00.431655 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:00.431631    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a247d001-e5aa-44f0-9213-b76f6e02dc64-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-a48099-29891880-r4lkn\" (UID: \"a247d001-e5aa-44f0-9213-b76f6e02dc64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a48099-29891880-r4lkn"
Nov 01 06:00:00.431759 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:00.431704    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd5575b5-1e5c-4224-8cc0-6a91d0445ddc-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891880-dkwgt\" (UID: \"bd5575b5-1e5c-4224-8cc0-6a91d0445ddc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891880-dkwgt"
Nov 01 06:00:00.490485 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:00.490432    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6chp\" (UniqueName: \"kubernetes.io/projected/69fb0a82-951c-43eb-9005-66798d4712b3-kube-api-access-r6chp\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-897c87-29891880-49f7n\" (UID: \"69fb0a82-951c-43eb-9005-66798d4712b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-897c87-29891880-49f7n"
Nov 01 06:00:00.490677 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:00.490656    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkmmb\" (UniqueName: \"kubernetes.io/projected/a247d001-e5aa-44f0-9213-b76f6e02dc64-kube-api-access-pkmmb\") pod \"maas-api-key-cleanup-e2e-worker-w4-a48099-29891880-r4lkn\" (UID: \"a247d001-e5aa-44f0-9213-b76f6e02dc64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a48099-29891880-r4lkn"
Nov 01 06:00:00.491356 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:00.491334    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r25p2\" (UniqueName: \"kubernetes.io/projected/bd5575b5-1e5c-4224-8cc0-6a91d0445ddc-kube-api-access-r25p2\") pod \"maas-api-key-cleanup-29891880-dkwgt\" (UID: \"bd5575b5-1e5c-4224-8cc0-6a91d0445ddc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891880-dkwgt"
Nov 01 06:00:00.512409 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:00.512385    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a48099-29891880-r4lkn"
Nov 01 06:00:00.529961 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:00.529940    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-897c87-29891880-49f7n"
Nov 01 06:00:00.537223 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:00.537202    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891880-dkwgt"
Nov 01 06:00:00.895394 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:00.895327    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a48099-29891880-r4lkn"]
Nov 01 06:00:00.916517 ip-10-0-129-92 kubenswrapper[5689]: W1101 06:00:00.916492    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda247d001_e5aa_44f0_9213_b76f6e02dc64.slice/crio-048991d5b9684dc4135b411facf98e4750eedd226eca0df4cdc86bd3a9bf5855 WatchSource:0}: Error finding container 048991d5b9684dc4135b411facf98e4750eedd226eca0df4cdc86bd3a9bf5855: Status 404 returned error can't find the container with id 048991d5b9684dc4135b411facf98e4750eedd226eca0df4cdc86bd3a9bf5855
Nov 01 06:00:00.946285 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:00.946261    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891880-dkwgt"]
Nov 01 06:00:00.952904 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:00.952881    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-897c87-29891880-49f7n"]
Nov 01 06:00:01.666037 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:01.666006    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-897c87-29891880-49f7n" event={"ID":"69fb0a82-951c-43eb-9005-66798d4712b3","Type":"ContainerStarted","Data":"abdd34c90f3fc4ee4fc0a7a627ac70d4b5080701caf66fbec3bf80473d3a270a"}
Nov 01 06:00:01.666208 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:01.666097    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-897c87-29891880-49f7n" event={"ID":"69fb0a82-951c-43eb-9005-66798d4712b3","Type":"ContainerStarted","Data":"83f58a37a3792da95fd9dbd3b34357ae66405139a98af19a39d97a441ef93e05"}
Nov 01 06:00:01.667645 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:01.667614    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a48099-29891880-r4lkn" event={"ID":"a247d001-e5aa-44f0-9213-b76f6e02dc64","Type":"ContainerStarted","Data":"cdffe2653b6bad32ce1c9cf211e2e32ef959c777537b64a11048ef9b09f9c9aa"}
Nov 01 06:00:01.667796 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:01.667661    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a48099-29891880-r4lkn" event={"ID":"a247d001-e5aa-44f0-9213-b76f6e02dc64","Type":"ContainerStarted","Data":"048991d5b9684dc4135b411facf98e4750eedd226eca0df4cdc86bd3a9bf5855"}
Nov 01 06:00:01.669114 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:01.669089    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891880-dkwgt" event={"ID":"bd5575b5-1e5c-4224-8cc0-6a91d0445ddc","Type":"ContainerStarted","Data":"3393adff154f68ef7ad8157673fed0fc947ffd957751fb2570a0fa76ec191e24"}
Nov 01 06:00:01.669232 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:01.669128    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891880-dkwgt" event={"ID":"bd5575b5-1e5c-4224-8cc0-6a91d0445ddc","Type":"ContainerStarted","Data":"94801c388359933c73a875a644aae9e5c03170469818800527a7bfba33fe11f9"}
Nov 01 06:00:01.781395 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:01.781343    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-897c87-29891880-49f7n" podStartSLOduration=1.781326049 podStartE2EDuration="1.781326049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:00:01.779914627 +0000 UTC m=+3379.850711350" watchObservedRunningTime="2026-11-01 06:00:01.781326049 +0000 UTC m=+3379.852122770"
Nov 01 06:00:01.896736 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:01.896686    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891880-dkwgt" podStartSLOduration=1.896670362 podStartE2EDuration="1.896670362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:00:01.895769515 +0000 UTC m=+3379.966566236" watchObservedRunningTime="2026-11-01 06:00:01.896670362 +0000 UTC m=+3379.967467082"
Nov 01 06:00:01.997013 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:01.996988    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg"
Nov 01 06:00:02.012021 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:02.011968    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a48099-29891880-r4lkn" podStartSLOduration=2.011951509 podStartE2EDuration="2.011951509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:00:02.009270965 +0000 UTC m=+3380.080067685" watchObservedRunningTime="2026-11-01 06:00:02.011951509 +0000 UTC m=+3380.082748228"
Nov 01 06:00:02.052106 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:02.052081    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba32daa8-34ac-44b9-a090-d4804cdeed41-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba32daa8-34ac-44b9-a090-d4804cdeed41-credential-socket\") pod \"ba32daa8-34ac-44b9-a090-d4804cdeed41\" (UID: \"ba32daa8-34ac-44b9-a090-d4804cdeed41\") "
Nov 01 06:00:02.052266 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:02.052147    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba32daa8-34ac-44b9-a090-d4804cdeed41-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba32daa8-34ac-44b9-a090-d4804cdeed41-workload-certs\") pod \"ba32daa8-34ac-44b9-a090-d4804cdeed41\" (UID: \"ba32daa8-34ac-44b9-a090-d4804cdeed41\") "
Nov 01 06:00:02.052266 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:02.052180    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba32daa8-34ac-44b9-a090-d4804cdeed41-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba32daa8-34ac-44b9-a090-d4804cdeed41-istio-envoy\") pod \"ba32daa8-34ac-44b9-a090-d4804cdeed41\" (UID: \"ba32daa8-34ac-44b9-a090-d4804cdeed41\") "
Nov 01 06:00:02.052266 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:02.052241    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba32daa8-34ac-44b9-a090-d4804cdeed41-kube-api-access-4nkng\" (UniqueName: \"kubernetes.io/projected/ba32daa8-34ac-44b9-a090-d4804cdeed41-kube-api-access-4nkng\") pod \"ba32daa8-34ac-44b9-a090-d4804cdeed41\" (UID: \"ba32daa8-34ac-44b9-a090-d4804cdeed41\") "
Nov 01 06:00:02.052433 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:02.052299    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba32daa8-34ac-44b9-a090-d4804cdeed41-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba32daa8-34ac-44b9-a090-d4804cdeed41-workload-socket\") pod \"ba32daa8-34ac-44b9-a090-d4804cdeed41\" (UID: \"ba32daa8-34ac-44b9-a090-d4804cdeed41\") "
Nov 01 06:00:02.052433 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:02.052353    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba32daa8-34ac-44b9-a090-d4804cdeed41-credential-socket" pod "ba32daa8-34ac-44b9-a090-d4804cdeed41" (UID: "ba32daa8-34ac-44b9-a090-d4804cdeed41"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:00:02.052433 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:02.052366    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba32daa8-34ac-44b9-a090-d4804cdeed41-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba32daa8-34ac-44b9-a090-d4804cdeed41-istiod-ca-cert\") pod \"ba32daa8-34ac-44b9-a090-d4804cdeed41\" (UID: \"ba32daa8-34ac-44b9-a090-d4804cdeed41\") "
Nov 01 06:00:02.052597 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:02.052439    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ba32daa8-34ac-44b9-a090-d4804cdeed41-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba32daa8-34ac-44b9-a090-d4804cdeed41-istio-podinfo\") pod \"ba32daa8-34ac-44b9-a090-d4804cdeed41\" (UID: \"ba32daa8-34ac-44b9-a090-d4804cdeed41\") "
Nov 01 06:00:02.052597 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:02.052482    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba32daa8-34ac-44b9-a090-d4804cdeed41-workload-certs" pod "ba32daa8-34ac-44b9-a090-d4804cdeed41" (UID: "ba32daa8-34ac-44b9-a090-d4804cdeed41"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:00:02.052597 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:02.052592    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba32daa8-34ac-44b9-a090-d4804cdeed41-istio-token\" (UniqueName: \"kubernetes.io/projected/ba32daa8-34ac-44b9-a090-d4804cdeed41-istio-token\") pod \"ba32daa8-34ac-44b9-a090-d4804cdeed41\" (UID: \"ba32daa8-34ac-44b9-a090-d4804cdeed41\") "
Nov 01 06:00:02.052751 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:02.052603    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba32daa8-34ac-44b9-a090-d4804cdeed41-workload-socket" pod "ba32daa8-34ac-44b9-a090-d4804cdeed41" (UID: "ba32daa8-34ac-44b9-a090-d4804cdeed41"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:00:02.052751 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:02.052639    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba32daa8-34ac-44b9-a090-d4804cdeed41-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba32daa8-34ac-44b9-a090-d4804cdeed41-istio-data\") pod \"ba32daa8-34ac-44b9-a090-d4804cdeed41\" (UID: \"ba32daa8-34ac-44b9-a090-d4804cdeed41\") "
Nov 01 06:00:02.052844 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:02.052793    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba32daa8-34ac-44b9-a090-d4804cdeed41-istiod-ca-cert" pod "ba32daa8-34ac-44b9-a090-d4804cdeed41" (UID: "ba32daa8-34ac-44b9-a090-d4804cdeed41"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:00:02.053096 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:02.053077    5689 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba32daa8-34ac-44b9-a090-d4804cdeed41-workload-socket\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:00:02.053169 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:02.053090    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba32daa8-34ac-44b9-a090-d4804cdeed41-istio-data" pod "ba32daa8-34ac-44b9-a090-d4804cdeed41" (UID: "ba32daa8-34ac-44b9-a090-d4804cdeed41"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:00:02.053169 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:02.053101    5689 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba32daa8-34ac-44b9-a090-d4804cdeed41-istiod-ca-cert\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:00:02.053169 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:02.053115    5689 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba32daa8-34ac-44b9-a090-d4804cdeed41-credential-socket\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:00:02.053169 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:02.053128    5689 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba32daa8-34ac-44b9-a090-d4804cdeed41-workload-certs\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:00:02.054587 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:02.054556    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba32daa8-34ac-44b9-a090-d4804cdeed41-kube-api-access-4nkng" pod "ba32daa8-34ac-44b9-a090-d4804cdeed41" (UID: "ba32daa8-34ac-44b9-a090-d4804cdeed41"). InnerVolumeSpecName "kube-api-access-4nkng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:00:02.054690 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:02.054649    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba32daa8-34ac-44b9-a090-d4804cdeed41-istio-token" pod "ba32daa8-34ac-44b9-a090-d4804cdeed41" (UID: "ba32daa8-34ac-44b9-a090-d4804cdeed41"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:00:02.055033 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:02.055012    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba32daa8-34ac-44b9-a090-d4804cdeed41-istio-podinfo" pod "ba32daa8-34ac-44b9-a090-d4804cdeed41" (UID: "ba32daa8-34ac-44b9-a090-d4804cdeed41"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:00:02.055183 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:02.055161    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba32daa8-34ac-44b9-a090-d4804cdeed41-istio-envoy" pod "ba32daa8-34ac-44b9-a090-d4804cdeed41" (UID: "ba32daa8-34ac-44b9-a090-d4804cdeed41"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:00:02.154381 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:02.154341    5689 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba32daa8-34ac-44b9-a090-d4804cdeed41-istio-token\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:00:02.154381 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:02.154379    5689 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba32daa8-34ac-44b9-a090-d4804cdeed41-istio-data\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:00:02.154636 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:02.154395    5689 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba32daa8-34ac-44b9-a090-d4804cdeed41-istio-envoy\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:00:02.154636 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:02.154410    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nkng\" (UniqueName: \"kubernetes.io/projected/ba32daa8-34ac-44b9-a090-d4804cdeed41-kube-api-access-4nkng\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:00:02.154636 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:02.154425    5689 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba32daa8-34ac-44b9-a090-d4804cdeed41-istio-podinfo\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:00:02.676561 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:02.676525    5689 generic.go:356] "Generic (PLEG): container finished" podID="69fb0a82-951c-43eb-9005-66798d4712b3" containerID="abdd34c90f3fc4ee4fc0a7a627ac70d4b5080701caf66fbec3bf80473d3a270a" exitCode=60
Nov 01 06:00:02.676748 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:02.676613    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-897c87-29891880-49f7n" event={"ID":"69fb0a82-951c-43eb-9005-66798d4712b3","Type":"ContainerDied","Data":"abdd34c90f3fc4ee4fc0a7a627ac70d4b5080701caf66fbec3bf80473d3a270a"}
Nov 01 06:00:02.676887 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:02.676871    5689 scope.go:122] "RemoveContainer" containerID="abdd34c90f3fc4ee4fc0a7a627ac70d4b5080701caf66fbec3bf80473d3a270a"
Nov 01 06:00:02.678182 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:02.678162    5689 generic.go:356] "Generic (PLEG): container finished" podID="a247d001-e5aa-44f0-9213-b76f6e02dc64" containerID="cdffe2653b6bad32ce1c9cf211e2e32ef959c777537b64a11048ef9b09f9c9aa" exitCode=60
Nov 01 06:00:02.678303 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:02.678253    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a48099-29891880-r4lkn" event={"ID":"a247d001-e5aa-44f0-9213-b76f6e02dc64","Type":"ContainerDied","Data":"cdffe2653b6bad32ce1c9cf211e2e32ef959c777537b64a11048ef9b09f9c9aa"}
Nov 01 06:00:02.678515 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:02.678497    5689 scope.go:122] "RemoveContainer" containerID="cdffe2653b6bad32ce1c9cf211e2e32ef959c777537b64a11048ef9b09f9c9aa"
Nov 01 06:00:02.679973 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:02.679862    5689 generic.go:356] "Generic (PLEG): container finished" podID="bd5575b5-1e5c-4224-8cc0-6a91d0445ddc" containerID="3393adff154f68ef7ad8157673fed0fc947ffd957751fb2570a0fa76ec191e24" exitCode=60
Nov 01 06:00:02.679973 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:02.679915    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891880-dkwgt" event={"ID":"bd5575b5-1e5c-4224-8cc0-6a91d0445ddc","Type":"ContainerDied","Data":"3393adff154f68ef7ad8157673fed0fc947ffd957751fb2570a0fa76ec191e24"}
Nov 01 06:00:02.680235 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:02.680089    5689 scope.go:122] "RemoveContainer" containerID="3393adff154f68ef7ad8157673fed0fc947ffd957751fb2570a0fa76ec191e24"
Nov 01 06:00:02.681859 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:02.681839    5689 generic.go:356] "Generic (PLEG): container finished" podID="ba32daa8-34ac-44b9-a090-d4804cdeed41" containerID="4505d6a083ef4eea3dadeb0a38fdafc33b63545dd3bc826114ca10b7d9af344d" exitCode=0
Nov 01 06:00:02.681932 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:02.681884    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg"
Nov 01 06:00:02.681932 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:02.681908    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg" event={"ID":"ba32daa8-34ac-44b9-a090-d4804cdeed41","Type":"ContainerDied","Data":"4505d6a083ef4eea3dadeb0a38fdafc33b63545dd3bc826114ca10b7d9af344d"}
Nov 01 06:00:02.682016 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:02.681952    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg" event={"ID":"ba32daa8-34ac-44b9-a090-d4804cdeed41","Type":"ContainerDied","Data":"e68a3fbf6542d90733dc72e32ce38c427a72450ae53cfafcf43a6f0271790bb4"}
Nov 01 06:00:02.682016 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:02.681974    5689 scope.go:122] "RemoveContainer" containerID="4505d6a083ef4eea3dadeb0a38fdafc33b63545dd3bc826114ca10b7d9af344d"
Nov 01 06:00:02.702310 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:02.702283    5689 scope.go:122] "RemoveContainer" containerID="e20fe1cec4efa6ca8c184e93cc6b2de1917576e46459be9b2c80b3b9130645c5"
Nov 01 06:00:02.732089 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:02.732057    5689 scope.go:122] "RemoveContainer" containerID="4505d6a083ef4eea3dadeb0a38fdafc33b63545dd3bc826114ca10b7d9af344d"
Nov 01 06:00:02.732333 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:00:02.732317    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4505d6a083ef4eea3dadeb0a38fdafc33b63545dd3bc826114ca10b7d9af344d\": container with ID starting with 4505d6a083ef4eea3dadeb0a38fdafc33b63545dd3bc826114ca10b7d9af344d not found: ID does not exist" containerID="4505d6a083ef4eea3dadeb0a38fdafc33b63545dd3bc826114ca10b7d9af344d"
Nov 01 06:00:02.732400 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:02.732342    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4505d6a083ef4eea3dadeb0a38fdafc33b63545dd3bc826114ca10b7d9af344d"} err="failed to get container status \"4505d6a083ef4eea3dadeb0a38fdafc33b63545dd3bc826114ca10b7d9af344d\": rpc error: code = NotFound desc = could not find container \"4505d6a083ef4eea3dadeb0a38fdafc33b63545dd3bc826114ca10b7d9af344d\": container with ID starting with 4505d6a083ef4eea3dadeb0a38fdafc33b63545dd3bc826114ca10b7d9af344d not found: ID does not exist"
Nov 01 06:00:02.732400 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:02.732357    5689 scope.go:122] "RemoveContainer" containerID="e20fe1cec4efa6ca8c184e93cc6b2de1917576e46459be9b2c80b3b9130645c5"
Nov 01 06:00:02.732590 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:00:02.732574    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e20fe1cec4efa6ca8c184e93cc6b2de1917576e46459be9b2c80b3b9130645c5\": container with ID starting with e20fe1cec4efa6ca8c184e93cc6b2de1917576e46459be9b2c80b3b9130645c5 not found: ID does not exist" containerID="e20fe1cec4efa6ca8c184e93cc6b2de1917576e46459be9b2c80b3b9130645c5"
Nov 01 06:00:02.732633 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:02.732595    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e20fe1cec4efa6ca8c184e93cc6b2de1917576e46459be9b2c80b3b9130645c5"} err="failed to get container status \"e20fe1cec4efa6ca8c184e93cc6b2de1917576e46459be9b2c80b3b9130645c5\": rpc error: code = NotFound desc = could not find container \"e20fe1cec4efa6ca8c184e93cc6b2de1917576e46459be9b2c80b3b9130645c5\": container with ID starting with e20fe1cec4efa6ca8c184e93cc6b2de1917576e46459be9b2c80b3b9130645c5 not found: ID does not exist"
Nov 01 06:00:03.131708 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:03.131676    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg"]
Nov 01 06:00:03.148333 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:03.148302    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-3d70ff-openshift-default-7c7dc9f4fc-sgrjg"]
Nov 01 06:00:03.689544 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:03.689516    5689 generic.go:356] "Generic (PLEG): container finished" podID="69fb0a82-951c-43eb-9005-66798d4712b3" containerID="830320617c323f32e9e515ae145d0c01f0c376fc717e0a06e6aded05382a5b07" exitCode=60
Nov 01 06:00:03.689681 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:03.689601    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-897c87-29891880-49f7n" event={"ID":"69fb0a82-951c-43eb-9005-66798d4712b3","Type":"ContainerDied","Data":"830320617c323f32e9e515ae145d0c01f0c376fc717e0a06e6aded05382a5b07"}
Nov 01 06:00:03.689681 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:03.689668    5689 scope.go:122] "RemoveContainer" containerID="abdd34c90f3fc4ee4fc0a7a627ac70d4b5080701caf66fbec3bf80473d3a270a"
Nov 01 06:00:03.689958 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:03.689940    5689 scope.go:122] "RemoveContainer" containerID="830320617c323f32e9e515ae145d0c01f0c376fc717e0a06e6aded05382a5b07"
Nov 01 06:00:03.690255 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:00:03.690226    5689 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-897c87-29891880-49f7n_odh-ai-gateway-infra(69fb0a82-951c-43eb-9005-66798d4712b3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-897c87-29891880-49f7n" podUID="69fb0a82-951c-43eb-9005-66798d4712b3"
Nov 01 06:00:03.691570 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:03.691551    5689 generic.go:356] "Generic (PLEG): container finished" podID="a247d001-e5aa-44f0-9213-b76f6e02dc64" containerID="464730e4a237ad1eed4d1da0fe74d4473a828966ef81277205806233e85f6667" exitCode=60
Nov 01 06:00:03.691682 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:03.691630    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a48099-29891880-r4lkn" event={"ID":"a247d001-e5aa-44f0-9213-b76f6e02dc64","Type":"ContainerDied","Data":"464730e4a237ad1eed4d1da0fe74d4473a828966ef81277205806233e85f6667"}
Nov 01 06:00:03.691852 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:03.691838    5689 scope.go:122] "RemoveContainer" containerID="464730e4a237ad1eed4d1da0fe74d4473a828966ef81277205806233e85f6667"
Nov 01 06:00:03.692100 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:00:03.692083    5689 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-a48099-29891880-r4lkn_odh-ai-gateway-infra(a247d001-e5aa-44f0-9213-b76f6e02dc64)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a48099-29891880-r4lkn" podUID="a247d001-e5aa-44f0-9213-b76f6e02dc64"
Nov 01 06:00:03.693474 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:03.693423    5689 generic.go:356] "Generic (PLEG): container finished" podID="bd5575b5-1e5c-4224-8cc0-6a91d0445ddc" containerID="a7feba7758a44490e1a455e789cb3ed1b5c71e707d9992ded29c53ffd828ca68" exitCode=60
Nov 01 06:00:03.693574 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:03.693497    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891880-dkwgt" event={"ID":"bd5575b5-1e5c-4224-8cc0-6a91d0445ddc","Type":"ContainerDied","Data":"a7feba7758a44490e1a455e789cb3ed1b5c71e707d9992ded29c53ffd828ca68"}
Nov 01 06:00:03.693758 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:03.693742    5689 scope.go:122] "RemoveContainer" containerID="a7feba7758a44490e1a455e789cb3ed1b5c71e707d9992ded29c53ffd828ca68"
Nov 01 06:00:03.694005 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:00:03.693981    5689 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891880-dkwgt_odh-ai-gateway-infra(bd5575b5-1e5c-4224-8cc0-6a91d0445ddc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891880-dkwgt" podUID="bd5575b5-1e5c-4224-8cc0-6a91d0445ddc"
Nov 01 06:00:03.755384 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:03.755366    5689 scope.go:122] "RemoveContainer" containerID="cdffe2653b6bad32ce1c9cf211e2e32ef959c777537b64a11048ef9b09f9c9aa"
Nov 01 06:00:03.767897 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:03.767874    5689 scope.go:122] "RemoveContainer" containerID="3393adff154f68ef7ad8157673fed0fc947ffd957751fb2570a0fa76ec191e24"
Nov 01 06:00:04.068529 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:04.068499    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba32daa8-34ac-44b9-a090-d4804cdeed41" path="/var/lib/kubelet/pods/ba32daa8-34ac-44b9-a090-d4804cdeed41/volumes"
Nov 01 06:00:04.504303 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:04.504272    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:00:04.703233 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:04.703206    5689 scope.go:122] "RemoveContainer" containerID="464730e4a237ad1eed4d1da0fe74d4473a828966ef81277205806233e85f6667"
Nov 01 06:00:04.703605 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:00:04.703576    5689 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-a48099-29891880-r4lkn_odh-ai-gateway-infra(a247d001-e5aa-44f0-9213-b76f6e02dc64)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a48099-29891880-r4lkn" podUID="a247d001-e5aa-44f0-9213-b76f6e02dc64"
Nov 01 06:00:04.704849 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:04.704828    5689 scope.go:122] "RemoveContainer" containerID="a7feba7758a44490e1a455e789cb3ed1b5c71e707d9992ded29c53ffd828ca68"
Nov 01 06:00:04.705124 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:00:04.705094    5689 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891880-dkwgt_odh-ai-gateway-infra(bd5575b5-1e5c-4224-8cc0-6a91d0445ddc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891880-dkwgt" podUID="bd5575b5-1e5c-4224-8cc0-6a91d0445ddc"
Nov 01 06:00:04.706640 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:04.706621    5689 scope.go:122] "RemoveContainer" containerID="830320617c323f32e9e515ae145d0c01f0c376fc717e0a06e6aded05382a5b07"
Nov 01 06:00:04.706847 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:00:04.706831    5689 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-897c87-29891880-49f7n_odh-ai-gateway-infra(69fb0a82-951c-43eb-9005-66798d4712b3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-897c87-29891880-49f7n" podUID="69fb0a82-951c-43eb-9005-66798d4712b3"
Nov 01 06:00:09.220999 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:09.220967    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:00:09.997865 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:09.997829    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:00:10.176259 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:10.176228    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:00:13.016072 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:13.016023    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:00:13.947673 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:13.947642    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:00:14.061281 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:14.061248    5689 scope.go:122] "RemoveContainer" containerID="a7feba7758a44490e1a455e789cb3ed1b5c71e707d9992ded29c53ffd828ca68"
Nov 01 06:00:14.793498 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:14.793446    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891880-dkwgt" event={"ID":"bd5575b5-1e5c-4224-8cc0-6a91d0445ddc","Type":"ContainerStarted","Data":"46185026ea74cce897b47b4a4f19a1be7568b6264b404d20b3f833e9e7ba8a4b"}
Nov 01 06:00:15.801314 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:15.801285    5689 generic.go:356] "Generic (PLEG): container finished" podID="bd5575b5-1e5c-4224-8cc0-6a91d0445ddc" containerID="46185026ea74cce897b47b4a4f19a1be7568b6264b404d20b3f833e9e7ba8a4b" exitCode=60
Nov 01 06:00:15.801763 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:15.801357    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891880-dkwgt" event={"ID":"bd5575b5-1e5c-4224-8cc0-6a91d0445ddc","Type":"ContainerDied","Data":"46185026ea74cce897b47b4a4f19a1be7568b6264b404d20b3f833e9e7ba8a4b"}
Nov 01 06:00:15.801763 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:15.801415    5689 scope.go:122] "RemoveContainer" containerID="a7feba7758a44490e1a455e789cb3ed1b5c71e707d9992ded29c53ffd828ca68"
Nov 01 06:00:15.801763 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:15.801726    5689 scope.go:122] "RemoveContainer" containerID="46185026ea74cce897b47b4a4f19a1be7568b6264b404d20b3f833e9e7ba8a4b"
Nov 01 06:00:15.801993 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:00:15.801975    5689 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29891880-dkwgt_odh-ai-gateway-infra(bd5575b5-1e5c-4224-8cc0-6a91d0445ddc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891880-dkwgt" podUID="bd5575b5-1e5c-4224-8cc0-6a91d0445ddc"
Nov 01 06:00:15.917339 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:15.917313    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891880-dkwgt"]
Nov 01 06:00:16.951817 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:16.951790    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891880-dkwgt"
Nov 01 06:00:17.044185 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:17.044152    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd5575b5-1e5c-4224-8cc0-6a91d0445ddc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd5575b5-1e5c-4224-8cc0-6a91d0445ddc-maas-api-service-ca\") pod \"bd5575b5-1e5c-4224-8cc0-6a91d0445ddc\" (UID: \"bd5575b5-1e5c-4224-8cc0-6a91d0445ddc\") "
Nov 01 06:00:17.044331 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:17.044311    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd5575b5-1e5c-4224-8cc0-6a91d0445ddc-kube-api-access-r25p2\" (UniqueName: \"kubernetes.io/projected/bd5575b5-1e5c-4224-8cc0-6a91d0445ddc-kube-api-access-r25p2\") pod \"bd5575b5-1e5c-4224-8cc0-6a91d0445ddc\" (UID: \"bd5575b5-1e5c-4224-8cc0-6a91d0445ddc\") "
Nov 01 06:00:17.044559 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:17.044537    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd5575b5-1e5c-4224-8cc0-6a91d0445ddc-maas-api-service-ca" pod "bd5575b5-1e5c-4224-8cc0-6a91d0445ddc" (UID: "bd5575b5-1e5c-4224-8cc0-6a91d0445ddc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:00:17.044649 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:17.044634    5689 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd5575b5-1e5c-4224-8cc0-6a91d0445ddc-maas-api-service-ca\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:00:17.046487 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:17.046469    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd5575b5-1e5c-4224-8cc0-6a91d0445ddc-kube-api-access-r25p2" pod "bd5575b5-1e5c-4224-8cc0-6a91d0445ddc" (UID: "bd5575b5-1e5c-4224-8cc0-6a91d0445ddc"). InnerVolumeSpecName "kube-api-access-r25p2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:00:17.061017 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:17.060974    5689 scope.go:122] "RemoveContainer" containerID="464730e4a237ad1eed4d1da0fe74d4473a828966ef81277205806233e85f6667"
Nov 01 06:00:17.146177 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:17.146148    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r25p2\" (UniqueName: \"kubernetes.io/projected/bd5575b5-1e5c-4224-8cc0-6a91d0445ddc-kube-api-access-r25p2\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:00:17.818435 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:17.818397    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a48099-29891880-r4lkn" event={"ID":"a247d001-e5aa-44f0-9213-b76f6e02dc64","Type":"ContainerStarted","Data":"8b907dfb4b4f491ed521dff54347cedeb01bc17ba33834e32294a71c91064252"}
Nov 01 06:00:17.819926 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:17.819900    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891880-dkwgt" event={"ID":"bd5575b5-1e5c-4224-8cc0-6a91d0445ddc","Type":"ContainerDied","Data":"94801c388359933c73a875a644aae9e5c03170469818800527a7bfba33fe11f9"}
Nov 01 06:00:17.819926 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:17.819925    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891880-dkwgt"
Nov 01 06:00:17.820123 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:17.819947    5689 scope.go:122] "RemoveContainer" containerID="46185026ea74cce897b47b4a4f19a1be7568b6264b404d20b3f833e9e7ba8a4b"
Nov 01 06:00:18.047167 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:18.047138    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891880-dkwgt"]
Nov 01 06:00:18.051015 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:18.050984    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891880-dkwgt"]
Nov 01 06:00:18.061411 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:18.061388    5689 scope.go:122] "RemoveContainer" containerID="830320617c323f32e9e515ae145d0c01f0c376fc717e0a06e6aded05382a5b07"
Nov 01 06:00:18.069918 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:18.069870    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd5575b5-1e5c-4224-8cc0-6a91d0445ddc" path="/var/lib/kubelet/pods/bd5575b5-1e5c-4224-8cc0-6a91d0445ddc/volumes"
Nov 01 06:00:18.803922 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:18.803892    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:00:18.838103 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:18.838062    5689 generic.go:356] "Generic (PLEG): container finished" podID="a247d001-e5aa-44f0-9213-b76f6e02dc64" containerID="8b907dfb4b4f491ed521dff54347cedeb01bc17ba33834e32294a71c91064252" exitCode=60
Nov 01 06:00:18.838417 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:18.838193    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a48099-29891880-r4lkn" event={"ID":"a247d001-e5aa-44f0-9213-b76f6e02dc64","Type":"ContainerDied","Data":"8b907dfb4b4f491ed521dff54347cedeb01bc17ba33834e32294a71c91064252"}
Nov 01 06:00:18.838417 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:18.838251    5689 scope.go:122] "RemoveContainer" containerID="464730e4a237ad1eed4d1da0fe74d4473a828966ef81277205806233e85f6667"
Nov 01 06:00:18.838795 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:18.838774    5689 scope.go:122] "RemoveContainer" containerID="8b907dfb4b4f491ed521dff54347cedeb01bc17ba33834e32294a71c91064252"
Nov 01 06:00:18.839141 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:00:18.839114    5689 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-a48099-29891880-r4lkn_odh-ai-gateway-infra(a247d001-e5aa-44f0-9213-b76f6e02dc64)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a48099-29891880-r4lkn" podUID="a247d001-e5aa-44f0-9213-b76f6e02dc64"
Nov 01 06:00:18.848560 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:18.848530    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-897c87-29891880-49f7n" event={"ID":"69fb0a82-951c-43eb-9005-66798d4712b3","Type":"ContainerStarted","Data":"db08f7917cf23b1d1cabcccd54dfee60927068dad5af9f8ed521e49fcf45fbee"}
Nov 01 06:00:18.946346 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:18.946316    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a48099-29891880-r4lkn"]
Nov 01 06:00:19.855921 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:19.855887    5689 generic.go:356] "Generic (PLEG): container finished" podID="69fb0a82-951c-43eb-9005-66798d4712b3" containerID="db08f7917cf23b1d1cabcccd54dfee60927068dad5af9f8ed521e49fcf45fbee" exitCode=60
Nov 01 06:00:19.856371 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:19.855959    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-897c87-29891880-49f7n" event={"ID":"69fb0a82-951c-43eb-9005-66798d4712b3","Type":"ContainerDied","Data":"db08f7917cf23b1d1cabcccd54dfee60927068dad5af9f8ed521e49fcf45fbee"}
Nov 01 06:00:19.856371 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:19.856032    5689 scope.go:122] "RemoveContainer" containerID="830320617c323f32e9e515ae145d0c01f0c376fc717e0a06e6aded05382a5b07"
Nov 01 06:00:19.856495 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:19.856385    5689 scope.go:122] "RemoveContainer" containerID="db08f7917cf23b1d1cabcccd54dfee60927068dad5af9f8ed521e49fcf45fbee"
Nov 01 06:00:19.856707 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:00:19.856683    5689 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-897c87-29891880-49f7n_odh-ai-gateway-infra(69fb0a82-951c-43eb-9005-66798d4712b3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-897c87-29891880-49f7n" podUID="69fb0a82-951c-43eb-9005-66798d4712b3"
Nov 01 06:00:19.998397 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:19.998376    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a48099-29891880-r4lkn"
Nov 01 06:00:20.075934 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:20.075903    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a247d001-e5aa-44f0-9213-b76f6e02dc64-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a247d001-e5aa-44f0-9213-b76f6e02dc64-maas-api-service-ca\") pod \"a247d001-e5aa-44f0-9213-b76f6e02dc64\" (UID: \"a247d001-e5aa-44f0-9213-b76f6e02dc64\") "
Nov 01 06:00:20.075934 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:20.075934    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a247d001-e5aa-44f0-9213-b76f6e02dc64-kube-api-access-pkmmb\" (UniqueName: \"kubernetes.io/projected/a247d001-e5aa-44f0-9213-b76f6e02dc64-kube-api-access-pkmmb\") pod \"a247d001-e5aa-44f0-9213-b76f6e02dc64\" (UID: \"a247d001-e5aa-44f0-9213-b76f6e02dc64\") "
Nov 01 06:00:20.076309 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:20.076285    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a247d001-e5aa-44f0-9213-b76f6e02dc64-maas-api-service-ca" pod "a247d001-e5aa-44f0-9213-b76f6e02dc64" (UID: "a247d001-e5aa-44f0-9213-b76f6e02dc64"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:00:20.078287 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:20.078250    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a247d001-e5aa-44f0-9213-b76f6e02dc64-kube-api-access-pkmmb" pod "a247d001-e5aa-44f0-9213-b76f6e02dc64" (UID: "a247d001-e5aa-44f0-9213-b76f6e02dc64"). InnerVolumeSpecName "kube-api-access-pkmmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:00:20.106163 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:20.106096    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-897c87-29891880-49f7n"]
Nov 01 06:00:20.177228 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:20.177195    5689 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a247d001-e5aa-44f0-9213-b76f6e02dc64-maas-api-service-ca\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:00:20.177228 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:20.177220    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkmmb\" (UniqueName: \"kubernetes.io/projected/a247d001-e5aa-44f0-9213-b76f6e02dc64-kube-api-access-pkmmb\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:00:20.865324 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:20.865295    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a48099-29891880-r4lkn"
Nov 01 06:00:20.865826 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:20.865329    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a48099-29891880-r4lkn" event={"ID":"a247d001-e5aa-44f0-9213-b76f6e02dc64","Type":"ContainerDied","Data":"048991d5b9684dc4135b411facf98e4750eedd226eca0df4cdc86bd3a9bf5855"}
Nov 01 06:00:20.865826 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:20.865390    5689 scope.go:122] "RemoveContainer" containerID="8b907dfb4b4f491ed521dff54347cedeb01bc17ba33834e32294a71c91064252"
Nov 01 06:00:21.007706 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:21.007681    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-897c87-29891880-49f7n"
Nov 01 06:00:21.031012 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:21.030985    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a48099-29891880-r4lkn"]
Nov 01 06:00:21.048111 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:21.048087    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a48099-29891880-r4lkn"]
Nov 01 06:00:21.087630 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:21.087603    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69fb0a82-951c-43eb-9005-66798d4712b3-kube-api-access-r6chp\" (UniqueName: \"kubernetes.io/projected/69fb0a82-951c-43eb-9005-66798d4712b3-kube-api-access-r6chp\") pod \"69fb0a82-951c-43eb-9005-66798d4712b3\" (UID: \"69fb0a82-951c-43eb-9005-66798d4712b3\") "
Nov 01 06:00:21.087746 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:21.087643    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69fb0a82-951c-43eb-9005-66798d4712b3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69fb0a82-951c-43eb-9005-66798d4712b3-maas-api-service-ca\") pod \"69fb0a82-951c-43eb-9005-66798d4712b3\" (UID: \"69fb0a82-951c-43eb-9005-66798d4712b3\") "
Nov 01 06:00:21.088030 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:21.088007    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69fb0a82-951c-43eb-9005-66798d4712b3-maas-api-service-ca" pod "69fb0a82-951c-43eb-9005-66798d4712b3" (UID: "69fb0a82-951c-43eb-9005-66798d4712b3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:00:21.089859 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:21.089838    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69fb0a82-951c-43eb-9005-66798d4712b3-kube-api-access-r6chp" pod "69fb0a82-951c-43eb-9005-66798d4712b3" (UID: "69fb0a82-951c-43eb-9005-66798d4712b3"). InnerVolumeSpecName "kube-api-access-r6chp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:00:21.189444 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:21.189411    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6chp\" (UniqueName: \"kubernetes.io/projected/69fb0a82-951c-43eb-9005-66798d4712b3-kube-api-access-r6chp\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:00:21.189602 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:21.189465    5689 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69fb0a82-951c-43eb-9005-66798d4712b3-maas-api-service-ca\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:00:21.876662 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:21.876626    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-897c87-29891880-49f7n" event={"ID":"69fb0a82-951c-43eb-9005-66798d4712b3","Type":"ContainerDied","Data":"83f58a37a3792da95fd9dbd3b34357ae66405139a98af19a39d97a441ef93e05"}
Nov 01 06:00:21.876662 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:21.876666    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-897c87-29891880-49f7n"
Nov 01 06:00:21.877159 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:21.876693    5689 scope.go:122] "RemoveContainer" containerID="db08f7917cf23b1d1cabcccd54dfee60927068dad5af9f8ed521e49fcf45fbee"
Nov 01 06:00:22.040565 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:22.040529    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-897c87-29891880-49f7n"]
Nov 01 06:00:22.047437 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:22.047411    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-897c87-29891880-49f7n"]
Nov 01 06:00:22.067254 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:22.067222    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69fb0a82-951c-43eb-9005-66798d4712b3" path="/var/lib/kubelet/pods/69fb0a82-951c-43eb-9005-66798d4712b3/volumes"
Nov 01 06:00:22.067658 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:22.067644    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a247d001-e5aa-44f0-9213-b76f6e02dc64" path="/var/lib/kubelet/pods/a247d001-e5aa-44f0-9213-b76f6e02dc64/volumes"
Nov 01 06:00:23.633774 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:23.633745    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:00:28.347958 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:28.347920    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:00:30.069617 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:00:30.069579    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 06:00:33.099160 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:33.099126    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:00:33.190588 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:33.190529    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:00:37.208814 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:37.208776    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-fchpl"]
Nov 01 06:00:37.209307 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:37.209294    5689 cpu_manager.go:414] "containerMap: removing container" podUID="69fb0a82-951c-43eb-9005-66798d4712b3" containerName="cleanup"
Nov 01 06:00:37.209362 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:37.209309    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69fb0a82-951c-43eb-9005-66798d4712b3" containerName="cleanup"
Nov 01 06:00:37.209362 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:37.209320    5689 cpu_manager.go:414] "containerMap: removing container" podUID="a247d001-e5aa-44f0-9213-b76f6e02dc64" containerName="cleanup"
Nov 01 06:00:37.209362 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:37.209325    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a247d001-e5aa-44f0-9213-b76f6e02dc64" containerName="cleanup"
Nov 01 06:00:37.209362 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:37.209336    5689 cpu_manager.go:414] "containerMap: removing container" podUID="bd5575b5-1e5c-4224-8cc0-6a91d0445ddc" containerName="cleanup"
Nov 01 06:00:37.209362 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:37.209341    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd5575b5-1e5c-4224-8cc0-6a91d0445ddc" containerName="cleanup"
Nov 01 06:00:37.209362 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:37.209356    5689 cpu_manager.go:414] "containerMap: removing container" podUID="a247d001-e5aa-44f0-9213-b76f6e02dc64" containerName="cleanup"
Nov 01 06:00:37.209362 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:37.209361    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a247d001-e5aa-44f0-9213-b76f6e02dc64" containerName="cleanup"
Nov 01 06:00:37.209633 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:37.209371    5689 cpu_manager.go:414] "containerMap: removing container" podUID="ba32daa8-34ac-44b9-a090-d4804cdeed41" containerName="istio-proxy"
Nov 01 06:00:37.209633 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:37.209377    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba32daa8-34ac-44b9-a090-d4804cdeed41" containerName="istio-proxy"
Nov 01 06:00:37.209633 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:37.209383    5689 cpu_manager.go:414] "containerMap: removing container" podUID="69fb0a82-951c-43eb-9005-66798d4712b3" containerName="cleanup"
Nov 01 06:00:37.209633 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:37.209388    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69fb0a82-951c-43eb-9005-66798d4712b3" containerName="cleanup"
Nov 01 06:00:37.209633 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:37.209395    5689 cpu_manager.go:414] "containerMap: removing container" podUID="bd5575b5-1e5c-4224-8cc0-6a91d0445ddc" containerName="cleanup"
Nov 01 06:00:37.209633 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:37.209398    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd5575b5-1e5c-4224-8cc0-6a91d0445ddc" containerName="cleanup"
Nov 01 06:00:37.209633 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:37.209407    5689 cpu_manager.go:414] "containerMap: removing container" podUID="69fb0a82-951c-43eb-9005-66798d4712b3" containerName="cleanup"
Nov 01 06:00:37.209633 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:37.209411    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69fb0a82-951c-43eb-9005-66798d4712b3" containerName="cleanup"
Nov 01 06:00:37.209633 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:37.209423    5689 cpu_manager.go:414] "containerMap: removing container" podUID="ba32daa8-34ac-44b9-a090-d4804cdeed41" containerName="istio-proxy"
Nov 01 06:00:37.209633 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:37.209427    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba32daa8-34ac-44b9-a090-d4804cdeed41" containerName="istio-proxy"
Nov 01 06:00:37.209633 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:37.209529    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="69fb0a82-951c-43eb-9005-66798d4712b3" containerName="cleanup"
Nov 01 06:00:37.209633 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:37.209538    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="a247d001-e5aa-44f0-9213-b76f6e02dc64" containerName="cleanup"
Nov 01 06:00:37.209633 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:37.209544    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba32daa8-34ac-44b9-a090-d4804cdeed41" containerName="istio-proxy"
Nov 01 06:00:37.209633 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:37.209551    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd5575b5-1e5c-4224-8cc0-6a91d0445ddc" containerName="cleanup"
Nov 01 06:00:37.209633 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:37.209561    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="69fb0a82-951c-43eb-9005-66798d4712b3" containerName="cleanup"
Nov 01 06:00:37.209633 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:37.209566    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="a247d001-e5aa-44f0-9213-b76f6e02dc64" containerName="cleanup"
Nov 01 06:00:37.209633 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:37.209571    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd5575b5-1e5c-4224-8cc0-6a91d0445ddc" containerName="cleanup"
Nov 01 06:00:37.209633 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:37.209578    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="a247d001-e5aa-44f0-9213-b76f6e02dc64" containerName="cleanup"
Nov 01 06:00:37.214342 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:37.214319    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-fchpl"
Nov 01 06:00:37.315807 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:37.315776    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-a48099-openshift-default-dockercfg-8mhxp\""
Nov 01 06:00:37.318952 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:37.318926    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-fchpl"]
Nov 01 06:00:37.371997 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:37.371966    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9001d49f-9167-48e7-bacd-91219748d5e1-istio-podinfo\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-fchpl\" (UID: \"9001d49f-9167-48e7-bacd-91219748d5e1\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-fchpl"
Nov 01 06:00:37.372152 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:37.372016    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9001d49f-9167-48e7-bacd-91219748d5e1-istio-data\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-fchpl\" (UID: \"9001d49f-9167-48e7-bacd-91219748d5e1\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-fchpl"
Nov 01 06:00:37.372204 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:37.372156    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9001d49f-9167-48e7-bacd-91219748d5e1-istio-envoy\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-fchpl\" (UID: \"9001d49f-9167-48e7-bacd-91219748d5e1\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-fchpl"
Nov 01 06:00:37.372204 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:37.372200    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9001d49f-9167-48e7-bacd-91219748d5e1-workload-certs\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-fchpl\" (UID: \"9001d49f-9167-48e7-bacd-91219748d5e1\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-fchpl"
Nov 01 06:00:37.372278 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:37.372225    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9001d49f-9167-48e7-bacd-91219748d5e1-workload-socket\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-fchpl\" (UID: \"9001d49f-9167-48e7-bacd-91219748d5e1\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-fchpl"
Nov 01 06:00:37.372324 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:37.372305    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9001d49f-9167-48e7-bacd-91219748d5e1-istio-token\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-fchpl\" (UID: \"9001d49f-9167-48e7-bacd-91219748d5e1\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-fchpl"
Nov 01 06:00:37.372360 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:37.372340    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9001d49f-9167-48e7-bacd-91219748d5e1-credential-socket\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-fchpl\" (UID: \"9001d49f-9167-48e7-bacd-91219748d5e1\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-fchpl"
Nov 01 06:00:37.372486 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:37.372448    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9001d49f-9167-48e7-bacd-91219748d5e1-istiod-ca-cert\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-fchpl\" (UID: \"9001d49f-9167-48e7-bacd-91219748d5e1\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-fchpl"
Nov 01 06:00:37.372568 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:37.372503    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cdgf\" (UniqueName: \"kubernetes.io/projected/9001d49f-9167-48e7-bacd-91219748d5e1-kube-api-access-7cdgf\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-fchpl\" (UID: \"9001d49f-9167-48e7-bacd-91219748d5e1\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-fchpl"
Nov 01 06:00:37.473475 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:37.473382    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9001d49f-9167-48e7-bacd-91219748d5e1-istio-envoy\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-fchpl\" (UID: \"9001d49f-9167-48e7-bacd-91219748d5e1\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-fchpl"
Nov 01 06:00:37.473475 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:37.473419    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9001d49f-9167-48e7-bacd-91219748d5e1-workload-certs\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-fchpl\" (UID: \"9001d49f-9167-48e7-bacd-91219748d5e1\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-fchpl"
Nov 01 06:00:37.473694 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:37.473476    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9001d49f-9167-48e7-bacd-91219748d5e1-workload-socket\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-fchpl\" (UID: \"9001d49f-9167-48e7-bacd-91219748d5e1\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-fchpl"
Nov 01 06:00:37.473694 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:37.473519    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9001d49f-9167-48e7-bacd-91219748d5e1-istio-token\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-fchpl\" (UID: \"9001d49f-9167-48e7-bacd-91219748d5e1\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-fchpl"
Nov 01 06:00:37.473694 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:37.473552    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9001d49f-9167-48e7-bacd-91219748d5e1-credential-socket\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-fchpl\" (UID: \"9001d49f-9167-48e7-bacd-91219748d5e1\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-fchpl"
Nov 01 06:00:37.473694 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:37.473605    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9001d49f-9167-48e7-bacd-91219748d5e1-istiod-ca-cert\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-fchpl\" (UID: \"9001d49f-9167-48e7-bacd-91219748d5e1\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-fchpl"
Nov 01 06:00:37.473694 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:37.473634    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cdgf\" (UniqueName: \"kubernetes.io/projected/9001d49f-9167-48e7-bacd-91219748d5e1-kube-api-access-7cdgf\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-fchpl\" (UID: \"9001d49f-9167-48e7-bacd-91219748d5e1\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-fchpl"
Nov 01 06:00:37.473694 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:37.473683    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9001d49f-9167-48e7-bacd-91219748d5e1-istio-podinfo\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-fchpl\" (UID: \"9001d49f-9167-48e7-bacd-91219748d5e1\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-fchpl"
Nov 01 06:00:37.473983 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:37.473740    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9001d49f-9167-48e7-bacd-91219748d5e1-istio-data\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-fchpl\" (UID: \"9001d49f-9167-48e7-bacd-91219748d5e1\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-fchpl"
Nov 01 06:00:37.473983 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:37.473915    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9001d49f-9167-48e7-bacd-91219748d5e1-workload-socket\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-fchpl\" (UID: \"9001d49f-9167-48e7-bacd-91219748d5e1\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-fchpl"
Nov 01 06:00:37.473983 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:37.473978    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9001d49f-9167-48e7-bacd-91219748d5e1-credential-socket\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-fchpl\" (UID: \"9001d49f-9167-48e7-bacd-91219748d5e1\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-fchpl"
Nov 01 06:00:37.474098 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:37.474042    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9001d49f-9167-48e7-bacd-91219748d5e1-istio-data\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-fchpl\" (UID: \"9001d49f-9167-48e7-bacd-91219748d5e1\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-fchpl"
Nov 01 06:00:37.474516 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:37.474151    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9001d49f-9167-48e7-bacd-91219748d5e1-workload-certs\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-fchpl\" (UID: \"9001d49f-9167-48e7-bacd-91219748d5e1\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-fchpl"
Nov 01 06:00:37.474516 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:37.474336    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9001d49f-9167-48e7-bacd-91219748d5e1-istiod-ca-cert\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-fchpl\" (UID: \"9001d49f-9167-48e7-bacd-91219748d5e1\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-fchpl"
Nov 01 06:00:37.475826 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:37.475809    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9001d49f-9167-48e7-bacd-91219748d5e1-istio-envoy\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-fchpl\" (UID: \"9001d49f-9167-48e7-bacd-91219748d5e1\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-fchpl"
Nov 01 06:00:37.476194 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:37.476176    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9001d49f-9167-48e7-bacd-91219748d5e1-istio-podinfo\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-fchpl\" (UID: \"9001d49f-9167-48e7-bacd-91219748d5e1\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-fchpl"
Nov 01 06:00:37.535604 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:37.535555    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9001d49f-9167-48e7-bacd-91219748d5e1-istio-token\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-fchpl\" (UID: \"9001d49f-9167-48e7-bacd-91219748d5e1\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-fchpl"
Nov 01 06:00:37.535717 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:37.535664    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cdgf\" (UniqueName: \"kubernetes.io/projected/9001d49f-9167-48e7-bacd-91219748d5e1-kube-api-access-7cdgf\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-fchpl\" (UID: \"9001d49f-9167-48e7-bacd-91219748d5e1\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-fchpl"
Nov 01 06:00:37.624686 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:37.624635    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fd2672-fd4cfd8c5-zs46l"]
Nov 01 06:00:37.625284 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:37.625266    5689 cpu_manager.go:414] "containerMap: removing container" podUID="a247d001-e5aa-44f0-9213-b76f6e02dc64" containerName="cleanup"
Nov 01 06:00:37.625363 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:37.625287    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a247d001-e5aa-44f0-9213-b76f6e02dc64" containerName="cleanup"
Nov 01 06:00:37.625363 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:37.625314    5689 cpu_manager.go:414] "containerMap: removing container" podUID="bd5575b5-1e5c-4224-8cc0-6a91d0445ddc" containerName="cleanup"
Nov 01 06:00:37.625363 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:37.625319    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd5575b5-1e5c-4224-8cc0-6a91d0445ddc" containerName="cleanup"
Nov 01 06:00:37.625502 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:37.625410    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="69fb0a82-951c-43eb-9005-66798d4712b3" containerName="cleanup"
Nov 01 06:00:37.625502 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:37.625421    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba32daa8-34ac-44b9-a090-d4804cdeed41" containerName="istio-proxy"
Nov 01 06:00:37.625502 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:37.625426    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd5575b5-1e5c-4224-8cc0-6a91d0445ddc" containerName="cleanup"
Nov 01 06:00:37.629999 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:37.629967    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fd2672-fd4cfd8c5-zs46l"
Nov 01 06:00:37.684574 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:37.684534    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-fd2672-serving-cert\""
Nov 01 06:00:37.687836 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:37.687813    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fd2672-fd4cfd8c5-zs46l"]
Nov 01 06:00:37.777515 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:37.777396    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9w8p\" (UniqueName: \"kubernetes.io/projected/f9e983e7-105f-4706-8872-3b3893f748f0-kube-api-access-l9w8p\") pod \"maas-api-e2e-shared-b-w1-fd2672-fd4cfd8c5-zs46l\" (UID: \"f9e983e7-105f-4706-8872-3b3893f748f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fd2672-fd4cfd8c5-zs46l"
Nov 01 06:00:37.777515 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:37.777483    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9e983e7-105f-4706-8872-3b3893f748f0-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-fd2672-fd4cfd8c5-zs46l\" (UID: \"f9e983e7-105f-4706-8872-3b3893f748f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fd2672-fd4cfd8c5-zs46l"
Nov 01 06:00:37.777742 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:37.777714    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9e983e7-105f-4706-8872-3b3893f748f0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-fd2672-fd4cfd8c5-zs46l\" (UID: \"f9e983e7-105f-4706-8872-3b3893f748f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fd2672-fd4cfd8c5-zs46l"
Nov 01 06:00:37.777833 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:37.777818    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9e983e7-105f-4706-8872-3b3893f748f0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-fd2672-fd4cfd8c5-zs46l\" (UID: \"f9e983e7-105f-4706-8872-3b3893f748f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fd2672-fd4cfd8c5-zs46l"
Nov 01 06:00:37.828783 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:37.828754    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-fchpl"
Nov 01 06:00:37.849888 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:37.849864    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:00:37.882818 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:37.882764    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9e983e7-105f-4706-8872-3b3893f748f0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-fd2672-fd4cfd8c5-zs46l\" (UID: \"f9e983e7-105f-4706-8872-3b3893f748f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fd2672-fd4cfd8c5-zs46l"
Nov 01 06:00:37.883013 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:37.882867    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9w8p\" (UniqueName: \"kubernetes.io/projected/f9e983e7-105f-4706-8872-3b3893f748f0-kube-api-access-l9w8p\") pod \"maas-api-e2e-shared-b-w1-fd2672-fd4cfd8c5-zs46l\" (UID: \"f9e983e7-105f-4706-8872-3b3893f748f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fd2672-fd4cfd8c5-zs46l"
Nov 01 06:00:37.883013 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:37.882898    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9e983e7-105f-4706-8872-3b3893f748f0-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-fd2672-fd4cfd8c5-zs46l\" (UID: \"f9e983e7-105f-4706-8872-3b3893f748f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fd2672-fd4cfd8c5-zs46l"
Nov 01 06:00:37.883180 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:37.883071    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9e983e7-105f-4706-8872-3b3893f748f0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-fd2672-fd4cfd8c5-zs46l\" (UID: \"f9e983e7-105f-4706-8872-3b3893f748f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fd2672-fd4cfd8c5-zs46l"
Nov 01 06:00:37.884315 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:37.884287    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9e983e7-105f-4706-8872-3b3893f748f0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-fd2672-fd4cfd8c5-zs46l\" (UID: \"f9e983e7-105f-4706-8872-3b3893f748f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fd2672-fd4cfd8c5-zs46l"
Nov 01 06:00:37.884495 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:00:37.884409    5689 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fd2672-serving-cert: secret "maas-api-e2e-shared-b-w1-fd2672-serving-cert" not found
Nov 01 06:00:37.884567 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:00:37.884508    5689 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9e983e7-105f-4706-8872-3b3893f748f0-maas-api-tls podName:f9e983e7-105f-4706-8872-3b3893f748f0 nodeName:}" failed. No retries permitted until 2026-11-01 06:00:38.384485445 +0000 UTC m=+3416.455282144 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f9e983e7-105f-4706-8872-3b3893f748f0-maas-api-tls") pod "maas-api-e2e-shared-b-w1-fd2672-fd4cfd8c5-zs46l" (UID: "f9e983e7-105f-4706-8872-3b3893f748f0") : secret "maas-api-e2e-shared-b-w1-fd2672-serving-cert" not found
Nov 01 06:00:37.887966 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:37.887907    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9e983e7-105f-4706-8872-3b3893f748f0-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-fd2672-fd4cfd8c5-zs46l\" (UID: \"f9e983e7-105f-4706-8872-3b3893f748f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fd2672-fd4cfd8c5-zs46l"
Nov 01 06:00:37.947995 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:37.947782    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9w8p\" (UniqueName: \"kubernetes.io/projected/f9e983e7-105f-4706-8872-3b3893f748f0-kube-api-access-l9w8p\") pod \"maas-api-e2e-shared-b-w1-fd2672-fd4cfd8c5-zs46l\" (UID: \"f9e983e7-105f-4706-8872-3b3893f748f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fd2672-fd4cfd8c5-zs46l"
Nov 01 06:00:38.051123 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:38.042903    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:00:38.229064 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:38.229009    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:00:38.309744 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:38.309720    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-fchpl"]
Nov 01 06:00:38.312250 ip-10-0-129-92 kubenswrapper[5689]: W1101 06:00:38.312225    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9001d49f_9167_48e7_bacd_91219748d5e1.slice/crio-c055d298a12200aec2d33e7cb33a97d90945f1239232924c74b82c353727a420 WatchSource:0}: Error finding container c055d298a12200aec2d33e7cb33a97d90945f1239232924c74b82c353727a420: Status 404 returned error can't find the container with id c055d298a12200aec2d33e7cb33a97d90945f1239232924c74b82c353727a420
Nov 01 06:00:38.314517 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:38.314484    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 06:00:38.314583 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:38.314553    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 06:00:38.314620 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:38.314582    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 06:00:38.388994 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:38.388973    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9e983e7-105f-4706-8872-3b3893f748f0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-fd2672-fd4cfd8c5-zs46l\" (UID: \"f9e983e7-105f-4706-8872-3b3893f748f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fd2672-fd4cfd8c5-zs46l"
Nov 01 06:00:38.391316 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:38.391291    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9e983e7-105f-4706-8872-3b3893f748f0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-fd2672-fd4cfd8c5-zs46l\" (UID: \"f9e983e7-105f-4706-8872-3b3893f748f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fd2672-fd4cfd8c5-zs46l"
Nov 01 06:00:38.544857 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:38.544824    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fd2672-fd4cfd8c5-zs46l"
Nov 01 06:00:38.751239 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:38.751197    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:00:38.927508 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:38.927442    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fd2672-fd4cfd8c5-zs46l"]
Nov 01 06:00:38.929939 ip-10-0-129-92 kubenswrapper[5689]: W1101 06:00:38.929909    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9e983e7_105f_4706_8872_3b3893f748f0.slice/crio-35812afc38b8574c33bebfcd3d097e43b53b66e74b036c93942b79488c601373 WatchSource:0}: Error finding container 35812afc38b8574c33bebfcd3d097e43b53b66e74b036c93942b79488c601373: Status 404 returned error can't find the container with id 35812afc38b8574c33bebfcd3d097e43b53b66e74b036c93942b79488c601373
Nov 01 06:00:38.931594 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:38.931576    5689 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 06:00:39.004422 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:39.004383    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fd2672-fd4cfd8c5-zs46l" event={"ID":"f9e983e7-105f-4706-8872-3b3893f748f0","Type":"ContainerStarted","Data":"35812afc38b8574c33bebfcd3d097e43b53b66e74b036c93942b79488c601373"}
Nov 01 06:00:39.006043 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:39.006015    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-fchpl" event={"ID":"9001d49f-9167-48e7-bacd-91219748d5e1","Type":"ContainerStarted","Data":"0062d22ba7f3f9526e542d6c2277e43627d76594563c663faa3e88bf44b73191"}
Nov 01 06:00:39.006172 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:39.006070    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-fchpl" event={"ID":"9001d49f-9167-48e7-bacd-91219748d5e1","Type":"ContainerStarted","Data":"c055d298a12200aec2d33e7cb33a97d90945f1239232924c74b82c353727a420"}
Nov 01 06:00:39.127396 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:39.127339    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-fchpl" podStartSLOduration=2.127323995 podStartE2EDuration="2.127323995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:00:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:00:39.125323344 +0000 UTC m=+3417.196120066" watchObservedRunningTime="2026-11-01 06:00:39.127323995 +0000 UTC m=+3417.198120715"
Nov 01 06:00:39.829668 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:39.829648    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-fchpl"
Nov 01 06:00:39.831619 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:39.831598    5689 patch_prober.go:29] interesting pod/e2e-worker-w4-a48099-openshift-default-577864759-fchpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body=
Nov 01 06:00:39.831721 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:39.831651    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-fchpl" podUID="9001d49f-9167-48e7-bacd-91219748d5e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused"
Nov 01 06:00:40.012820 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:40.012783    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fd2672-fd4cfd8c5-zs46l" event={"ID":"f9e983e7-105f-4706-8872-3b3893f748f0","Type":"ContainerStarted","Data":"dd6c68f4fc24183ad07b37f7394e968330d34a6a68c8117a4fcbe0f8925934e6"}
Nov 01 06:00:40.013003 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:40.012899    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fd2672-fd4cfd8c5-zs46l"
Nov 01 06:00:40.130003 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:40.129901    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fd2672-fd4cfd8c5-zs46l" podStartSLOduration=2.244820238 podStartE2EDuration="3.129887701s" totalImagesPullingTime="885.067463ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:00:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:00:40.129215588 +0000 UTC m=+3418.200012330" watchObservedRunningTime="2026-11-01 06:00:40.129887701 +0000 UTC m=+3418.200684418"
Nov 01 06:00:40.830204 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:40.830169    5689 patch_prober.go:29] interesting pod/e2e-worker-w4-a48099-openshift-default-577864759-fchpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body=
Nov 01 06:00:40.830724 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:40.830228    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-fchpl" podUID="9001d49f-9167-48e7-bacd-91219748d5e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused"
Nov 01 06:00:41.830071 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:41.830034    5689 patch_prober.go:29] interesting pod/e2e-worker-w4-a48099-openshift-default-577864759-fchpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body=
Nov 01 06:00:41.830273 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:41.830096    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-fchpl" podUID="9001d49f-9167-48e7-bacd-91219748d5e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused"
Nov 01 06:00:42.819031 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:42.818985    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:00:42.830071 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:42.830032    5689 patch_prober.go:29] interesting pod/e2e-worker-w4-a48099-openshift-default-577864759-fchpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body=
Nov 01 06:00:42.830358 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:42.830327    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-fchpl" podUID="9001d49f-9167-48e7-bacd-91219748d5e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused"
Nov 01 06:00:43.829601 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:43.829562    5689 patch_prober.go:29] interesting pod/e2e-worker-w4-a48099-openshift-default-577864759-fchpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body=
Nov 01 06:00:43.829784 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:43.829614    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-fchpl" podUID="9001d49f-9167-48e7-bacd-91219748d5e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused"
Nov 01 06:00:44.430243 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:44.430214    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:00:44.830085 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:44.830012    5689 patch_prober.go:29] interesting pod/e2e-worker-w4-a48099-openshift-default-577864759-fchpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body=
Nov 01 06:00:44.830085 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:44.830072    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-fchpl" podUID="9001d49f-9167-48e7-bacd-91219748d5e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused"
Nov 01 06:00:45.888116 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:45.888088    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-fchpl"
Nov 01 06:00:45.888719 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:45.888384    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-fchpl"
Nov 01 06:00:45.889447 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:45.889420    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-fchpl"
Nov 01 06:00:46.024745 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:46.024721    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fd2672-fd4cfd8c5-zs46l"
Nov 01 06:00:47.608400 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:47.608366    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:00:52.533149 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:52.533074    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:00:57.303810 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:00:57.303779    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:01:00.069057 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:01:00.069025    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 06:01:02.127556 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:02.127508    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:01:06.894666 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:06.894631    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:01:11.580126 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:11.580096    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:01:13.321244 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:13.321206    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:01:14.819607 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:14.819573    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:01:15.220994 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:15.220963    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c2zj8"]
Nov 01 06:01:15.221185 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:15.221145    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c2zj8" podUID="da1ad4a4-629e-4ce5-a336-1bf832074db4" containerName="istio-proxy" containerID="cri-o://089b3a823924d2a44720451f9f96d5a406e59222190c02ab126a83d3230504a9" gracePeriod=30
Nov 01 06:01:15.224196 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:15.224171    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vwd6x"]
Nov 01 06:01:15.224437 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:15.224401    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vwd6x" podUID="5b840b67-ec8d-45e0-91b4-0a8dc7a353eb" containerName="istio-proxy" containerID="cri-o://2b5c9cf467da138c68971f4e06576313f60d33f678dad045b4a19f60e06d260a" gracePeriod=30
Nov 01 06:01:15.270679 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:15.270649    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zml98"]
Nov 01 06:01:15.270909 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:15.270886    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zml98" podUID="758d79fd-92ca-46fb-b487-694dc9818b61" containerName="istio-proxy" containerID="cri-o://09a876b78766cd3fd778dbb4a32c11412f8beef5e1b7b9cb2e67eec381071c26" gracePeriod=30
Nov 01 06:01:15.272625 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:15.272603    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5ljv7"]
Nov 01 06:01:15.272807 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:15.272774    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5ljv7" podUID="82632280-5edc-47e8-a570-f98de80de625" containerName="istio-proxy" containerID="cri-o://b10ceda72be8c8b582ad06854fd96170f7d7778826dff56bc054ecd22b160dc0" gracePeriod=30
Nov 01 06:01:15.529123 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:15.522356    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-c2zj8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 06:01:15.554956 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:15.554918    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c2zj8" podUID="da1ad4a4-629e-4ce5-a336-1bf832074db4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 06:01:15.586479 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:15.585213    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-vwd6x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 06:01:15.586479 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:15.585288    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vwd6x" podUID="5b840b67-ec8d-45e0-91b4-0a8dc7a353eb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 06:01:16.329243 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:16.329208    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:01:17.579772 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:17.579730    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-zml98 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 06:01:17.580237 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:17.579794    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zml98" podUID="758d79fd-92ca-46fb-b487-694dc9818b61" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 06:01:20.560409 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.560381    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c2zj8"
Nov 01 06:01:20.587708 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.587678    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vwd6x"
Nov 01 06:01:20.605216 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.605192    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b840b67-ec8d-45e0-91b4-0a8dc7a353eb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b840b67-ec8d-45e0-91b4-0a8dc7a353eb-workload-certs\") pod \"5b840b67-ec8d-45e0-91b4-0a8dc7a353eb\" (UID: \"5b840b67-ec8d-45e0-91b4-0a8dc7a353eb\") "
Nov 01 06:01:20.605356 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.605279    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da1ad4a4-629e-4ce5-a336-1bf832074db4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da1ad4a4-629e-4ce5-a336-1bf832074db4-istiod-ca-cert\") pod \"da1ad4a4-629e-4ce5-a336-1bf832074db4\" (UID: \"da1ad4a4-629e-4ce5-a336-1bf832074db4\") "
Nov 01 06:01:20.605356 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.605315    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da1ad4a4-629e-4ce5-a336-1bf832074db4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da1ad4a4-629e-4ce5-a336-1bf832074db4-workload-certs\") pod \"da1ad4a4-629e-4ce5-a336-1bf832074db4\" (UID: \"da1ad4a4-629e-4ce5-a336-1bf832074db4\") "
Nov 01 06:01:20.605356 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.605339    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b840b67-ec8d-45e0-91b4-0a8dc7a353eb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b840b67-ec8d-45e0-91b4-0a8dc7a353eb-credential-socket\") pod \"5b840b67-ec8d-45e0-91b4-0a8dc7a353eb\" (UID: \"5b840b67-ec8d-45e0-91b4-0a8dc7a353eb\") "
Nov 01 06:01:20.605565 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.605465    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b840b67-ec8d-45e0-91b4-0a8dc7a353eb-workload-certs" pod "5b840b67-ec8d-45e0-91b4-0a8dc7a353eb" (UID: "5b840b67-ec8d-45e0-91b4-0a8dc7a353eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:01:20.605565 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.605555    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5b840b67-ec8d-45e0-91b4-0a8dc7a353eb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b840b67-ec8d-45e0-91b4-0a8dc7a353eb-istio-podinfo\") pod \"5b840b67-ec8d-45e0-91b4-0a8dc7a353eb\" (UID: \"5b840b67-ec8d-45e0-91b4-0a8dc7a353eb\") "
Nov 01 06:01:20.605662 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.605564    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da1ad4a4-629e-4ce5-a336-1bf832074db4-workload-certs" pod "da1ad4a4-629e-4ce5-a336-1bf832074db4" (UID: "da1ad4a4-629e-4ce5-a336-1bf832074db4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:01:20.605662 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.605591    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b840b67-ec8d-45e0-91b4-0a8dc7a353eb-istio-token\" (UniqueName: \"kubernetes.io/projected/5b840b67-ec8d-45e0-91b4-0a8dc7a353eb-istio-token\") pod \"5b840b67-ec8d-45e0-91b4-0a8dc7a353eb\" (UID: \"5b840b67-ec8d-45e0-91b4-0a8dc7a353eb\") "
Nov 01 06:01:20.605662 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.605640    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b840b67-ec8d-45e0-91b4-0a8dc7a353eb-credential-socket" pod "5b840b67-ec8d-45e0-91b4-0a8dc7a353eb" (UID: "5b840b67-ec8d-45e0-91b4-0a8dc7a353eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:01:20.605811 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.605659    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da1ad4a4-629e-4ce5-a336-1bf832074db4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da1ad4a4-629e-4ce5-a336-1bf832074db4-istio-envoy\") pod \"da1ad4a4-629e-4ce5-a336-1bf832074db4\" (UID: \"da1ad4a4-629e-4ce5-a336-1bf832074db4\") "
Nov 01 06:01:20.605811 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.605729    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da1ad4a4-629e-4ce5-a336-1bf832074db4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da1ad4a4-629e-4ce5-a336-1bf832074db4-istio-data\") pod \"da1ad4a4-629e-4ce5-a336-1bf832074db4\" (UID: \"da1ad4a4-629e-4ce5-a336-1bf832074db4\") "
Nov 01 06:01:20.605811 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.605771    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b840b67-ec8d-45e0-91b4-0a8dc7a353eb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b840b67-ec8d-45e0-91b4-0a8dc7a353eb-istio-envoy\") pod \"5b840b67-ec8d-45e0-91b4-0a8dc7a353eb\" (UID: \"5b840b67-ec8d-45e0-91b4-0a8dc7a353eb\") "
Nov 01 06:01:20.605811 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.605803    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b840b67-ec8d-45e0-91b4-0a8dc7a353eb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b840b67-ec8d-45e0-91b4-0a8dc7a353eb-istiod-ca-cert\") pod \"5b840b67-ec8d-45e0-91b4-0a8dc7a353eb\" (UID: \"5b840b67-ec8d-45e0-91b4-0a8dc7a353eb\") "
Nov 01 06:01:20.606016 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.605729    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da1ad4a4-629e-4ce5-a336-1bf832074db4-istiod-ca-cert" pod "da1ad4a4-629e-4ce5-a336-1bf832074db4" (UID: "da1ad4a4-629e-4ce5-a336-1bf832074db4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:01:20.606016 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.605883    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/da1ad4a4-629e-4ce5-a336-1bf832074db4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da1ad4a4-629e-4ce5-a336-1bf832074db4-istio-podinfo\") pod \"da1ad4a4-629e-4ce5-a336-1bf832074db4\" (UID: \"da1ad4a4-629e-4ce5-a336-1bf832074db4\") "
Nov 01 06:01:20.606016 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.605923    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da1ad4a4-629e-4ce5-a336-1bf832074db4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da1ad4a4-629e-4ce5-a336-1bf832074db4-workload-socket\") pod \"da1ad4a4-629e-4ce5-a336-1bf832074db4\" (UID: \"da1ad4a4-629e-4ce5-a336-1bf832074db4\") "
Nov 01 06:01:20.606016 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.605963    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da1ad4a4-629e-4ce5-a336-1bf832074db4-kube-api-access-mvfx9\" (UniqueName: \"kubernetes.io/projected/da1ad4a4-629e-4ce5-a336-1bf832074db4-kube-api-access-mvfx9\") pod \"da1ad4a4-629e-4ce5-a336-1bf832074db4\" (UID: \"da1ad4a4-629e-4ce5-a336-1bf832074db4\") "
Nov 01 06:01:20.607283 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.607257    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da1ad4a4-629e-4ce5-a336-1bf832074db4-istio-token\" (UniqueName: \"kubernetes.io/projected/da1ad4a4-629e-4ce5-a336-1bf832074db4-istio-token\") pod \"da1ad4a4-629e-4ce5-a336-1bf832074db4\" (UID: \"da1ad4a4-629e-4ce5-a336-1bf832074db4\") "
Nov 01 06:01:20.607397 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.607309    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da1ad4a4-629e-4ce5-a336-1bf832074db4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da1ad4a4-629e-4ce5-a336-1bf832074db4-credential-socket\") pod \"da1ad4a4-629e-4ce5-a336-1bf832074db4\" (UID: \"da1ad4a4-629e-4ce5-a336-1bf832074db4\") "
Nov 01 06:01:20.607397 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.607362    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b840b67-ec8d-45e0-91b4-0a8dc7a353eb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b840b67-ec8d-45e0-91b4-0a8dc7a353eb-istio-data\") pod \"5b840b67-ec8d-45e0-91b4-0a8dc7a353eb\" (UID: \"5b840b67-ec8d-45e0-91b4-0a8dc7a353eb\") "
Nov 01 06:01:20.607533 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.607401    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b840b67-ec8d-45e0-91b4-0a8dc7a353eb-kube-api-access-k27zz\" (UniqueName: \"kubernetes.io/projected/5b840b67-ec8d-45e0-91b4-0a8dc7a353eb-kube-api-access-k27zz\") pod \"5b840b67-ec8d-45e0-91b4-0a8dc7a353eb\" (UID: \"5b840b67-ec8d-45e0-91b4-0a8dc7a353eb\") "
Nov 01 06:01:20.607533 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.607428    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b840b67-ec8d-45e0-91b4-0a8dc7a353eb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b840b67-ec8d-45e0-91b4-0a8dc7a353eb-workload-socket\") pod \"5b840b67-ec8d-45e0-91b4-0a8dc7a353eb\" (UID: \"5b840b67-ec8d-45e0-91b4-0a8dc7a353eb\") "
Nov 01 06:01:20.608196 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.606057    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da1ad4a4-629e-4ce5-a336-1bf832074db4-istio-data" pod "da1ad4a4-629e-4ce5-a336-1bf832074db4" (UID: "da1ad4a4-629e-4ce5-a336-1bf832074db4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:01:20.608196 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.606780    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da1ad4a4-629e-4ce5-a336-1bf832074db4-workload-socket" pod "da1ad4a4-629e-4ce5-a336-1bf832074db4" (UID: "da1ad4a4-629e-4ce5-a336-1bf832074db4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:01:20.608347 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.607160    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b840b67-ec8d-45e0-91b4-0a8dc7a353eb-istiod-ca-cert" pod "5b840b67-ec8d-45e0-91b4-0a8dc7a353eb" (UID: "5b840b67-ec8d-45e0-91b4-0a8dc7a353eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:01:20.608347 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.608169    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da1ad4a4-629e-4ce5-a336-1bf832074db4-credential-socket" pod "da1ad4a4-629e-4ce5-a336-1bf832074db4" (UID: "da1ad4a4-629e-4ce5-a336-1bf832074db4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:01:20.608779 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.608580    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b840b67-ec8d-45e0-91b4-0a8dc7a353eb-istio-token" pod "5b840b67-ec8d-45e0-91b4-0a8dc7a353eb" (UID: "5b840b67-ec8d-45e0-91b4-0a8dc7a353eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:01:20.608779 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.608687    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5b840b67-ec8d-45e0-91b4-0a8dc7a353eb-istio-podinfo" pod "5b840b67-ec8d-45e0-91b4-0a8dc7a353eb" (UID: "5b840b67-ec8d-45e0-91b4-0a8dc7a353eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:01:20.608779 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.608713    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b840b67-ec8d-45e0-91b4-0a8dc7a353eb-istio-data" pod "5b840b67-ec8d-45e0-91b4-0a8dc7a353eb" (UID: "5b840b67-ec8d-45e0-91b4-0a8dc7a353eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:01:20.608779 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.608737    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da1ad4a4-629e-4ce5-a336-1bf832074db4-istio-envoy" pod "da1ad4a4-629e-4ce5-a336-1bf832074db4" (UID: "da1ad4a4-629e-4ce5-a336-1bf832074db4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:01:20.609023 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.608835    5689 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b840b67-ec8d-45e0-91b4-0a8dc7a353eb-istio-data\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:01:20.609023 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.608864    5689 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b840b67-ec8d-45e0-91b4-0a8dc7a353eb-workload-certs\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:01:20.609023 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.608879    5689 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da1ad4a4-629e-4ce5-a336-1bf832074db4-istiod-ca-cert\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:01:20.609023 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.608892    5689 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da1ad4a4-629e-4ce5-a336-1bf832074db4-workload-certs\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:01:20.609023 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.608888    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b840b67-ec8d-45e0-91b4-0a8dc7a353eb-workload-socket" pod "5b840b67-ec8d-45e0-91b4-0a8dc7a353eb" (UID: "5b840b67-ec8d-45e0-91b4-0a8dc7a353eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:01:20.609023 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.608904    5689 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b840b67-ec8d-45e0-91b4-0a8dc7a353eb-credential-socket\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:01:20.609023 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.608952    5689 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b840b67-ec8d-45e0-91b4-0a8dc7a353eb-istio-podinfo\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:01:20.609391 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.609071    5689 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b840b67-ec8d-45e0-91b4-0a8dc7a353eb-istio-token\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:01:20.609391 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.609099    5689 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da1ad4a4-629e-4ce5-a336-1bf832074db4-istio-envoy\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:01:20.609391 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.609112    5689 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da1ad4a4-629e-4ce5-a336-1bf832074db4-istio-data\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:01:20.609391 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.609124    5689 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b840b67-ec8d-45e0-91b4-0a8dc7a353eb-istiod-ca-cert\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:01:20.609391 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.609140    5689 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da1ad4a4-629e-4ce5-a336-1bf832074db4-workload-socket\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:01:20.609391 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.609155    5689 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da1ad4a4-629e-4ce5-a336-1bf832074db4-credential-socket\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:01:20.609802 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.609778    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da1ad4a4-629e-4ce5-a336-1bf832074db4-istio-podinfo" pod "da1ad4a4-629e-4ce5-a336-1bf832074db4" (UID: "da1ad4a4-629e-4ce5-a336-1bf832074db4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:01:20.610969 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.610943    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b840b67-ec8d-45e0-91b4-0a8dc7a353eb-istio-envoy" pod "5b840b67-ec8d-45e0-91b4-0a8dc7a353eb" (UID: "5b840b67-ec8d-45e0-91b4-0a8dc7a353eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:01:20.611063 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.610992    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b840b67-ec8d-45e0-91b4-0a8dc7a353eb-kube-api-access-k27zz" pod "5b840b67-ec8d-45e0-91b4-0a8dc7a353eb" (UID: "5b840b67-ec8d-45e0-91b4-0a8dc7a353eb"). InnerVolumeSpecName "kube-api-access-k27zz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:01:20.611063 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.611016    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da1ad4a4-629e-4ce5-a336-1bf832074db4-kube-api-access-mvfx9" pod "da1ad4a4-629e-4ce5-a336-1bf832074db4" (UID: "da1ad4a4-629e-4ce5-a336-1bf832074db4"). InnerVolumeSpecName "kube-api-access-mvfx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:01:20.611063 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.611046    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da1ad4a4-629e-4ce5-a336-1bf832074db4-istio-token" pod "da1ad4a4-629e-4ce5-a336-1bf832074db4" (UID: "da1ad4a4-629e-4ce5-a336-1bf832074db4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:01:20.632003 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.631976    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zml98"
Nov 01 06:01:20.634314 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.634297    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5ljv7"
Nov 01 06:01:20.709697 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.709671    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82632280-5edc-47e8-a570-f98de80de625-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82632280-5edc-47e8-a570-f98de80de625-istio-envoy\") pod \"82632280-5edc-47e8-a570-f98de80de625\" (UID: \"82632280-5edc-47e8-a570-f98de80de625\") "
Nov 01 06:01:20.709834 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.709714    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/758d79fd-92ca-46fb-b487-694dc9818b61-istio-token\" (UniqueName: \"kubernetes.io/projected/758d79fd-92ca-46fb-b487-694dc9818b61-istio-token\") pod \"758d79fd-92ca-46fb-b487-694dc9818b61\" (UID: \"758d79fd-92ca-46fb-b487-694dc9818b61\") "
Nov 01 06:01:20.709834 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.709734    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/82632280-5edc-47e8-a570-f98de80de625-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82632280-5edc-47e8-a570-f98de80de625-istio-podinfo\") pod \"82632280-5edc-47e8-a570-f98de80de625\" (UID: \"82632280-5edc-47e8-a570-f98de80de625\") "
Nov 01 06:01:20.709834 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.709774    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/758d79fd-92ca-46fb-b487-694dc9818b61-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/758d79fd-92ca-46fb-b487-694dc9818b61-istio-podinfo\") pod \"758d79fd-92ca-46fb-b487-694dc9818b61\" (UID: \"758d79fd-92ca-46fb-b487-694dc9818b61\") "
Nov 01 06:01:20.709834 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.709800    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/758d79fd-92ca-46fb-b487-694dc9818b61-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/758d79fd-92ca-46fb-b487-694dc9818b61-istiod-ca-cert\") pod \"758d79fd-92ca-46fb-b487-694dc9818b61\" (UID: \"758d79fd-92ca-46fb-b487-694dc9818b61\") "
Nov 01 06:01:20.710061 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.709837    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82632280-5edc-47e8-a570-f98de80de625-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82632280-5edc-47e8-a570-f98de80de625-credential-socket\") pod \"82632280-5edc-47e8-a570-f98de80de625\" (UID: \"82632280-5edc-47e8-a570-f98de80de625\") "
Nov 01 06:01:20.710061 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.709980    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82632280-5edc-47e8-a570-f98de80de625-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82632280-5edc-47e8-a570-f98de80de625-workload-certs\") pod \"82632280-5edc-47e8-a570-f98de80de625\" (UID: \"82632280-5edc-47e8-a570-f98de80de625\") "
Nov 01 06:01:20.710170 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.710060    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82632280-5edc-47e8-a570-f98de80de625-kube-api-access-rncn7\" (UniqueName: \"kubernetes.io/projected/82632280-5edc-47e8-a570-f98de80de625-kube-api-access-rncn7\") pod \"82632280-5edc-47e8-a570-f98de80de625\" (UID: \"82632280-5edc-47e8-a570-f98de80de625\") "
Nov 01 06:01:20.710170 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.710105    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/758d79fd-92ca-46fb-b487-694dc9818b61-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/758d79fd-92ca-46fb-b487-694dc9818b61-workload-socket\") pod \"758d79fd-92ca-46fb-b487-694dc9818b61\" (UID: \"758d79fd-92ca-46fb-b487-694dc9818b61\") "
Nov 01 06:01:20.710170 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.710124    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82632280-5edc-47e8-a570-f98de80de625-credential-socket" pod "82632280-5edc-47e8-a570-f98de80de625" (UID: "82632280-5edc-47e8-a570-f98de80de625"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:01:20.710170 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.710151    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/758d79fd-92ca-46fb-b487-694dc9818b61-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/758d79fd-92ca-46fb-b487-694dc9818b61-istio-envoy\") pod \"758d79fd-92ca-46fb-b487-694dc9818b61\" (UID: \"758d79fd-92ca-46fb-b487-694dc9818b61\") "
Nov 01 06:01:20.710394 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.710185    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82632280-5edc-47e8-a570-f98de80de625-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82632280-5edc-47e8-a570-f98de80de625-istiod-ca-cert\") pod \"82632280-5edc-47e8-a570-f98de80de625\" (UID: \"82632280-5edc-47e8-a570-f98de80de625\") "
Nov 01 06:01:20.710394 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.710219    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/758d79fd-92ca-46fb-b487-694dc9818b61-kube-api-access-4rfw4\" (UniqueName: \"kubernetes.io/projected/758d79fd-92ca-46fb-b487-694dc9818b61-kube-api-access-4rfw4\") pod \"758d79fd-92ca-46fb-b487-694dc9818b61\" (UID: \"758d79fd-92ca-46fb-b487-694dc9818b61\") "
Nov 01 06:01:20.710394 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.710264    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82632280-5edc-47e8-a570-f98de80de625-workload-certs" pod "82632280-5edc-47e8-a570-f98de80de625" (UID: "82632280-5edc-47e8-a570-f98de80de625"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:01:20.710394 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.710271    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82632280-5edc-47e8-a570-f98de80de625-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82632280-5edc-47e8-a570-f98de80de625-workload-socket\") pod \"82632280-5edc-47e8-a570-f98de80de625\" (UID: \"82632280-5edc-47e8-a570-f98de80de625\") "
Nov 01 06:01:20.710394 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.710283    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/758d79fd-92ca-46fb-b487-694dc9818b61-istiod-ca-cert" pod "758d79fd-92ca-46fb-b487-694dc9818b61" (UID: "758d79fd-92ca-46fb-b487-694dc9818b61"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:01:20.710394 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.710300    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/758d79fd-92ca-46fb-b487-694dc9818b61-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/758d79fd-92ca-46fb-b487-694dc9818b61-workload-certs\") pod \"758d79fd-92ca-46fb-b487-694dc9818b61\" (UID: \"758d79fd-92ca-46fb-b487-694dc9818b61\") "
Nov 01 06:01:20.710394 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.710330    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/758d79fd-92ca-46fb-b487-694dc9818b61-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/758d79fd-92ca-46fb-b487-694dc9818b61-istio-data\") pod \"758d79fd-92ca-46fb-b487-694dc9818b61\" (UID: \"758d79fd-92ca-46fb-b487-694dc9818b61\") "
Nov 01 06:01:20.710394 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.710373    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/758d79fd-92ca-46fb-b487-694dc9818b61-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/758d79fd-92ca-46fb-b487-694dc9818b61-credential-socket\") pod \"758d79fd-92ca-46fb-b487-694dc9818b61\" (UID: \"758d79fd-92ca-46fb-b487-694dc9818b61\") "
Nov 01 06:01:20.711032 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.710501    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82632280-5edc-47e8-a570-f98de80de625-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82632280-5edc-47e8-a570-f98de80de625-istio-data\") pod \"82632280-5edc-47e8-a570-f98de80de625\" (UID: \"82632280-5edc-47e8-a570-f98de80de625\") "
Nov 01 06:01:20.711032 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.710538    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82632280-5edc-47e8-a570-f98de80de625-istio-token\" (UniqueName: \"kubernetes.io/projected/82632280-5edc-47e8-a570-f98de80de625-istio-token\") pod \"82632280-5edc-47e8-a570-f98de80de625\" (UID: \"82632280-5edc-47e8-a570-f98de80de625\") "
Nov 01 06:01:20.711032 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.710684    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/758d79fd-92ca-46fb-b487-694dc9818b61-workload-socket" pod "758d79fd-92ca-46fb-b487-694dc9818b61" (UID: "758d79fd-92ca-46fb-b487-694dc9818b61"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:01:20.711208 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.711095    5689 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/758d79fd-92ca-46fb-b487-694dc9818b61-istiod-ca-cert\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:01:20.711208 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.711118    5689 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82632280-5edc-47e8-a570-f98de80de625-credential-socket\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:01:20.711208 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.711135    5689 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82632280-5edc-47e8-a570-f98de80de625-workload-certs\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:01:20.711208 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.711150    5689 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/758d79fd-92ca-46fb-b487-694dc9818b61-workload-socket\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:01:20.711208 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.711164    5689 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b840b67-ec8d-45e0-91b4-0a8dc7a353eb-istio-envoy\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:01:20.711208 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.711178    5689 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da1ad4a4-629e-4ce5-a336-1bf832074db4-istio-podinfo\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:01:20.711208 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.711193    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvfx9\" (UniqueName: \"kubernetes.io/projected/da1ad4a4-629e-4ce5-a336-1bf832074db4-kube-api-access-mvfx9\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:01:20.711208 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.711208    5689 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da1ad4a4-629e-4ce5-a336-1bf832074db4-istio-token\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:01:20.711651 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.711222    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k27zz\" (UniqueName: \"kubernetes.io/projected/5b840b67-ec8d-45e0-91b4-0a8dc7a353eb-kube-api-access-k27zz\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:01:20.711651 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.711236    5689 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b840b67-ec8d-45e0-91b4-0a8dc7a353eb-workload-socket\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:01:20.711651 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.711443    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/758d79fd-92ca-46fb-b487-694dc9818b61-credential-socket" pod "758d79fd-92ca-46fb-b487-694dc9818b61" (UID: "758d79fd-92ca-46fb-b487-694dc9818b61"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:01:20.711869 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.711709    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/758d79fd-92ca-46fb-b487-694dc9818b61-istio-data" pod "758d79fd-92ca-46fb-b487-694dc9818b61" (UID: "758d79fd-92ca-46fb-b487-694dc9818b61"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:01:20.711933 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.711909    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82632280-5edc-47e8-a570-f98de80de625-workload-socket" pod "82632280-5edc-47e8-a570-f98de80de625" (UID: "82632280-5edc-47e8-a570-f98de80de625"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:01:20.711984 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.711936    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/758d79fd-92ca-46fb-b487-694dc9818b61-workload-certs" pod "758d79fd-92ca-46fb-b487-694dc9818b61" (UID: "758d79fd-92ca-46fb-b487-694dc9818b61"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:01:20.712044 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.712008    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82632280-5edc-47e8-a570-f98de80de625-istio-data" pod "82632280-5edc-47e8-a570-f98de80de625" (UID: "82632280-5edc-47e8-a570-f98de80de625"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:01:20.712103 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.712088    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82632280-5edc-47e8-a570-f98de80de625-istiod-ca-cert" pod "82632280-5edc-47e8-a570-f98de80de625" (UID: "82632280-5edc-47e8-a570-f98de80de625"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:01:20.712759 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.712731    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/758d79fd-92ca-46fb-b487-694dc9818b61-istio-token" pod "758d79fd-92ca-46fb-b487-694dc9818b61" (UID: "758d79fd-92ca-46fb-b487-694dc9818b61"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:01:20.712882 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.712846    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/758d79fd-92ca-46fb-b487-694dc9818b61-istio-podinfo" pod "758d79fd-92ca-46fb-b487-694dc9818b61" (UID: "758d79fd-92ca-46fb-b487-694dc9818b61"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:01:20.712980 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.712958    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82632280-5edc-47e8-a570-f98de80de625-istio-envoy" pod "82632280-5edc-47e8-a570-f98de80de625" (UID: "82632280-5edc-47e8-a570-f98de80de625"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:01:20.713093 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.713071    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/82632280-5edc-47e8-a570-f98de80de625-istio-podinfo" pod "82632280-5edc-47e8-a570-f98de80de625" (UID: "82632280-5edc-47e8-a570-f98de80de625"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:01:20.713236 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.713151    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82632280-5edc-47e8-a570-f98de80de625-kube-api-access-rncn7" pod "82632280-5edc-47e8-a570-f98de80de625" (UID: "82632280-5edc-47e8-a570-f98de80de625"). InnerVolumeSpecName "kube-api-access-rncn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:01:20.713849 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.713829    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82632280-5edc-47e8-a570-f98de80de625-istio-token" pod "82632280-5edc-47e8-a570-f98de80de625" (UID: "82632280-5edc-47e8-a570-f98de80de625"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:01:20.714170 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.714155    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/758d79fd-92ca-46fb-b487-694dc9818b61-kube-api-access-4rfw4" pod "758d79fd-92ca-46fb-b487-694dc9818b61" (UID: "758d79fd-92ca-46fb-b487-694dc9818b61"). InnerVolumeSpecName "kube-api-access-4rfw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:01:20.714223 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.714168    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/758d79fd-92ca-46fb-b487-694dc9818b61-istio-envoy" pod "758d79fd-92ca-46fb-b487-694dc9818b61" (UID: "758d79fd-92ca-46fb-b487-694dc9818b61"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:01:20.812656 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.812587    5689 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82632280-5edc-47e8-a570-f98de80de625-istio-data\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:01:20.812656 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.812611    5689 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82632280-5edc-47e8-a570-f98de80de625-istio-token\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:01:20.812656 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.812620    5689 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82632280-5edc-47e8-a570-f98de80de625-istio-envoy\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:01:20.812656 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.812629    5689 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/758d79fd-92ca-46fb-b487-694dc9818b61-istio-token\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:01:20.812656 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.812638    5689 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82632280-5edc-47e8-a570-f98de80de625-istio-podinfo\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:01:20.812656 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.812647    5689 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/758d79fd-92ca-46fb-b487-694dc9818b61-istio-podinfo\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:01:20.812656 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.812655    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rncn7\" (UniqueName: \"kubernetes.io/projected/82632280-5edc-47e8-a570-f98de80de625-kube-api-access-rncn7\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:01:20.812971 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.812665    5689 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/758d79fd-92ca-46fb-b487-694dc9818b61-istio-envoy\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:01:20.812971 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.812675    5689 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82632280-5edc-47e8-a570-f98de80de625-istiod-ca-cert\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:01:20.812971 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.812683    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rfw4\" (UniqueName: \"kubernetes.io/projected/758d79fd-92ca-46fb-b487-694dc9818b61-kube-api-access-4rfw4\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:01:20.812971 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.812692    5689 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82632280-5edc-47e8-a570-f98de80de625-workload-socket\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:01:20.812971 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.812700    5689 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/758d79fd-92ca-46fb-b487-694dc9818b61-workload-certs\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:01:20.812971 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.812707    5689 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/758d79fd-92ca-46fb-b487-694dc9818b61-istio-data\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:01:20.812971 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:20.812715    5689 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/758d79fd-92ca-46fb-b487-694dc9818b61-credential-socket\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:01:21.077424 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:21.077358    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:01:21.262070 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:21.262037    5689 generic.go:356] "Generic (PLEG): container finished" podID="82632280-5edc-47e8-a570-f98de80de625" containerID="b10ceda72be8c8b582ad06854fd96170f7d7778826dff56bc054ecd22b160dc0" exitCode=0
Nov 01 06:01:21.262209 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:21.262092    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5ljv7"
Nov 01 06:01:21.262209 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:21.262091    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5ljv7" event={"ID":"82632280-5edc-47e8-a570-f98de80de625","Type":"ContainerDied","Data":"b10ceda72be8c8b582ad06854fd96170f7d7778826dff56bc054ecd22b160dc0"}
Nov 01 06:01:21.262304 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:21.262243    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5ljv7" event={"ID":"82632280-5edc-47e8-a570-f98de80de625","Type":"ContainerDied","Data":"1136ad74033d8b4d6a2e9efdad5cb0098412b703988e6044c9b5eaf573881a1b"}
Nov 01 06:01:21.262304 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:21.262284    5689 scope.go:122] "RemoveContainer" containerID="b10ceda72be8c8b582ad06854fd96170f7d7778826dff56bc054ecd22b160dc0"
Nov 01 06:01:21.263760 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:21.263738    5689 generic.go:356] "Generic (PLEG): container finished" podID="5b840b67-ec8d-45e0-91b4-0a8dc7a353eb" containerID="2b5c9cf467da138c68971f4e06576313f60d33f678dad045b4a19f60e06d260a" exitCode=0
Nov 01 06:01:21.263880 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:21.263819    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vwd6x"
Nov 01 06:01:21.263955 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:21.263881    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vwd6x" event={"ID":"5b840b67-ec8d-45e0-91b4-0a8dc7a353eb","Type":"ContainerDied","Data":"2b5c9cf467da138c68971f4e06576313f60d33f678dad045b4a19f60e06d260a"}
Nov 01 06:01:21.263955 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:21.263925    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vwd6x" event={"ID":"5b840b67-ec8d-45e0-91b4-0a8dc7a353eb","Type":"ContainerDied","Data":"42f0e3d432167355a397d4fe650a3bdf47ea99a21279b6fe0e4b88aa0f9afcaf"}
Nov 01 06:01:21.265818 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:21.265796    5689 generic.go:356] "Generic (PLEG): container finished" podID="758d79fd-92ca-46fb-b487-694dc9818b61" containerID="09a876b78766cd3fd778dbb4a32c11412f8beef5e1b7b9cb2e67eec381071c26" exitCode=0
Nov 01 06:01:21.265944 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:21.265840    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zml98"
Nov 01 06:01:21.265944 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:21.265878    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zml98" event={"ID":"758d79fd-92ca-46fb-b487-694dc9818b61","Type":"ContainerDied","Data":"09a876b78766cd3fd778dbb4a32c11412f8beef5e1b7b9cb2e67eec381071c26"}
Nov 01 06:01:21.265944 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:21.265927    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zml98" event={"ID":"758d79fd-92ca-46fb-b487-694dc9818b61","Type":"ContainerDied","Data":"cca0c60df1e519b9417f10a553d6b6386fe1fdcf4aa03ccba15d1dd59cca8ecc"}
Nov 01 06:01:21.269976 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:21.269806    5689 generic.go:356] "Generic (PLEG): container finished" podID="da1ad4a4-629e-4ce5-a336-1bf832074db4" containerID="089b3a823924d2a44720451f9f96d5a406e59222190c02ab126a83d3230504a9" exitCode=0
Nov 01 06:01:21.269976 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:21.269871    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c2zj8" event={"ID":"da1ad4a4-629e-4ce5-a336-1bf832074db4","Type":"ContainerDied","Data":"089b3a823924d2a44720451f9f96d5a406e59222190c02ab126a83d3230504a9"}
Nov 01 06:01:21.269976 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:21.269914    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c2zj8"
Nov 01 06:01:21.269976 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:21.269941    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c2zj8" event={"ID":"da1ad4a4-629e-4ce5-a336-1bf832074db4","Type":"ContainerDied","Data":"da105a73fe8677a10ae7bfb71fa1dde0e47bb089f4050bde43935bf961e84dd7"}
Nov 01 06:01:21.285536 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:21.285517    5689 scope.go:122] "RemoveContainer" containerID="b10ceda72be8c8b582ad06854fd96170f7d7778826dff56bc054ecd22b160dc0"
Nov 01 06:01:21.285807 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:01:21.285789    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b10ceda72be8c8b582ad06854fd96170f7d7778826dff56bc054ecd22b160dc0\": container with ID starting with b10ceda72be8c8b582ad06854fd96170f7d7778826dff56bc054ecd22b160dc0 not found: ID does not exist" containerID="b10ceda72be8c8b582ad06854fd96170f7d7778826dff56bc054ecd22b160dc0"
Nov 01 06:01:21.285858 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:21.285817    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b10ceda72be8c8b582ad06854fd96170f7d7778826dff56bc054ecd22b160dc0"} err="failed to get container status \"b10ceda72be8c8b582ad06854fd96170f7d7778826dff56bc054ecd22b160dc0\": rpc error: code = NotFound desc = could not find container \"b10ceda72be8c8b582ad06854fd96170f7d7778826dff56bc054ecd22b160dc0\": container with ID starting with b10ceda72be8c8b582ad06854fd96170f7d7778826dff56bc054ecd22b160dc0 not found: ID does not exist"
Nov 01 06:01:21.285858 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:21.285835    5689 scope.go:122] "RemoveContainer" containerID="2b5c9cf467da138c68971f4e06576313f60d33f678dad045b4a19f60e06d260a"
Nov 01 06:01:21.294631 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:21.294617    5689 scope.go:122] "RemoveContainer" containerID="2b5c9cf467da138c68971f4e06576313f60d33f678dad045b4a19f60e06d260a"
Nov 01 06:01:21.294907 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:01:21.294888    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b5c9cf467da138c68971f4e06576313f60d33f678dad045b4a19f60e06d260a\": container with ID starting with 2b5c9cf467da138c68971f4e06576313f60d33f678dad045b4a19f60e06d260a not found: ID does not exist" containerID="2b5c9cf467da138c68971f4e06576313f60d33f678dad045b4a19f60e06d260a"
Nov 01 06:01:21.294952 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:21.294913    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b5c9cf467da138c68971f4e06576313f60d33f678dad045b4a19f60e06d260a"} err="failed to get container status \"2b5c9cf467da138c68971f4e06576313f60d33f678dad045b4a19f60e06d260a\": rpc error: code = NotFound desc = could not find container \"2b5c9cf467da138c68971f4e06576313f60d33f678dad045b4a19f60e06d260a\": container with ID starting with 2b5c9cf467da138c68971f4e06576313f60d33f678dad045b4a19f60e06d260a not found: ID does not exist"
Nov 01 06:01:21.294952 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:21.294927    5689 scope.go:122] "RemoveContainer" containerID="09a876b78766cd3fd778dbb4a32c11412f8beef5e1b7b9cb2e67eec381071c26"
Nov 01 06:01:21.303220 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:21.303201    5689 scope.go:122] "RemoveContainer" containerID="09a876b78766cd3fd778dbb4a32c11412f8beef5e1b7b9cb2e67eec381071c26"
Nov 01 06:01:21.303483 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:01:21.303465    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09a876b78766cd3fd778dbb4a32c11412f8beef5e1b7b9cb2e67eec381071c26\": container with ID starting with 09a876b78766cd3fd778dbb4a32c11412f8beef5e1b7b9cb2e67eec381071c26 not found: ID does not exist" containerID="09a876b78766cd3fd778dbb4a32c11412f8beef5e1b7b9cb2e67eec381071c26"
Nov 01 06:01:21.303548 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:21.303488    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09a876b78766cd3fd778dbb4a32c11412f8beef5e1b7b9cb2e67eec381071c26"} err="failed to get container status \"09a876b78766cd3fd778dbb4a32c11412f8beef5e1b7b9cb2e67eec381071c26\": rpc error: code = NotFound desc = could not find container \"09a876b78766cd3fd778dbb4a32c11412f8beef5e1b7b9cb2e67eec381071c26\": container with ID starting with 09a876b78766cd3fd778dbb4a32c11412f8beef5e1b7b9cb2e67eec381071c26 not found: ID does not exist"
Nov 01 06:01:21.303548 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:21.303502    5689 scope.go:122] "RemoveContainer" containerID="089b3a823924d2a44720451f9f96d5a406e59222190c02ab126a83d3230504a9"
Nov 01 06:01:21.311823 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:21.311806    5689 scope.go:122] "RemoveContainer" containerID="089b3a823924d2a44720451f9f96d5a406e59222190c02ab126a83d3230504a9"
Nov 01 06:01:21.312099 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:01:21.312063    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"089b3a823924d2a44720451f9f96d5a406e59222190c02ab126a83d3230504a9\": container with ID starting with 089b3a823924d2a44720451f9f96d5a406e59222190c02ab126a83d3230504a9 not found: ID does not exist" containerID="089b3a823924d2a44720451f9f96d5a406e59222190c02ab126a83d3230504a9"
Nov 01 06:01:21.312099 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:21.312086    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"089b3a823924d2a44720451f9f96d5a406e59222190c02ab126a83d3230504a9"} err="failed to get container status \"089b3a823924d2a44720451f9f96d5a406e59222190c02ab126a83d3230504a9\": rpc error: code = NotFound desc = could not find container \"089b3a823924d2a44720451f9f96d5a406e59222190c02ab126a83d3230504a9\": container with ID starting with 089b3a823924d2a44720451f9f96d5a406e59222190c02ab126a83d3230504a9 not found: ID does not exist"
Nov 01 06:01:21.427015 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:21.426989    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vwd6x"]
Nov 01 06:01:21.433102 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:21.433082    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-vwd6x"]
Nov 01 06:01:21.541629 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:21.541601    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c2zj8"]
Nov 01 06:01:21.545688 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:21.545663    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c2zj8"]
Nov 01 06:01:21.653709 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:21.653681    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zml98"]
Nov 01 06:01:21.657717 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:21.657695    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-zml98"]
Nov 01 06:01:21.766288 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:21.766211    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5ljv7"]
Nov 01 06:01:21.771043 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:21.771014    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-5ljv7"]
Nov 01 06:01:22.067814 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:22.067743    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b840b67-ec8d-45e0-91b4-0a8dc7a353eb" path="/var/lib/kubelet/pods/5b840b67-ec8d-45e0-91b4-0a8dc7a353eb/volumes"
Nov 01 06:01:22.068194 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:22.068180    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="758d79fd-92ca-46fb-b487-694dc9818b61" path="/var/lib/kubelet/pods/758d79fd-92ca-46fb-b487-694dc9818b61/volumes"
Nov 01 06:01:22.068601 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:22.068588    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82632280-5edc-47e8-a570-f98de80de625" path="/var/lib/kubelet/pods/82632280-5edc-47e8-a570-f98de80de625/volumes"
Nov 01 06:01:22.069302 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:22.069286    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da1ad4a4-629e-4ce5-a336-1bf832074db4" path="/var/lib/kubelet/pods/da1ad4a4-629e-4ce5-a336-1bf832074db4/volumes"
Nov 01 06:01:25.797510 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:25.797445    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:01:26.446717 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:26.446664    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:01:27.085899 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:27.085869    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:01:27.998709 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:27.998673    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:01:28.454965 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:28.454932    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:01:30.068594 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:01:30.068558    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 06:01:30.611305 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:30.611270    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:01:31.790671 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:31.790640    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:01:31.960319 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:31.960287    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:01:33.104759 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:33.104729    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:01:34.506709 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:34.506679    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:01:35.446128 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:35.446082    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:01:36.090987 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:36.090953    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:01:36.357683 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:36.357608    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:01:40.225616 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:40.225579    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:01:40.489131 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:40.489050    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:01:42.358203 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:42.358175    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:01:44.975630 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:44.975594    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:01:45.156069 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:45.156033    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:01:45.333700 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:45.333615    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:01:45.733920 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:45.733854    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:01:50.073511 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:50.073426    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:01:51.849070 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:51.849029    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:01:52.029738 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:52.029707    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:01:52.201106 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:52.201070    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:01:52.376028 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:52.375996    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:01:52.795574 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:52.795541    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:01:54.830977 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:54.830945    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:01:59.076880 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:59.076846    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:01:59.258677 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:59.258632    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:01:59.455824 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:59.455789    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:01:59.636986 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:01:59.636953    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:02:00.061602 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:00.061566    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:02:00.069838 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:02:00.069782    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 06:02:00.236314 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:00.236284    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:02:01.796220 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:01.796187    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:02:02.008155 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:02.008127    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:02:04.402634 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:04.402602    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:02:05.344441 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:05.344406    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:02:06.049657 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:06.049618    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:02:06.222821 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:06.222784    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:02:11.008050 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:11.008023    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:02:13.364364 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:13.364329    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:02:13.543441 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:13.543404    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:02:13.723821 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:13.723788    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:02:15.107900 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:15.107867    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:02:15.692099 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:15.692062    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx"]
Nov 01 06:02:15.692348 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:15.692309    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx" podUID="9d69cb72-f4e4-4ef1-9ceb-210cbd671a54" containerName="istio-proxy" containerID="cri-o://7c9ff3969a214671a9128620947f7f6691626fcd9fc594c29d3568c4a79e9642" gracePeriod=30
Nov 01 06:02:15.780659 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:15.780625    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:02:15.959362 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:15.959280    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:02:16.521823 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:16.521792    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:02:17.832960 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:17.832925    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:02:20.954883 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:20.954822    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx"
Nov 01 06:02:21.009148 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:21.009115    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d69cb72-f4e4-4ef1-9ceb-210cbd671a54-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d69cb72-f4e4-4ef1-9ceb-210cbd671a54-istiod-ca-cert\") pod \"9d69cb72-f4e4-4ef1-9ceb-210cbd671a54\" (UID: \"9d69cb72-f4e4-4ef1-9ceb-210cbd671a54\") "
Nov 01 06:02:21.009311 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:21.009157    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d69cb72-f4e4-4ef1-9ceb-210cbd671a54-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d69cb72-f4e4-4ef1-9ceb-210cbd671a54-workload-certs\") pod \"9d69cb72-f4e4-4ef1-9ceb-210cbd671a54\" (UID: \"9d69cb72-f4e4-4ef1-9ceb-210cbd671a54\") "
Nov 01 06:02:21.009311 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:21.009247    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9d69cb72-f4e4-4ef1-9ceb-210cbd671a54-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d69cb72-f4e4-4ef1-9ceb-210cbd671a54-istio-podinfo\") pod \"9d69cb72-f4e4-4ef1-9ceb-210cbd671a54\" (UID: \"9d69cb72-f4e4-4ef1-9ceb-210cbd671a54\") "
Nov 01 06:02:21.009311 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:21.009294    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d69cb72-f4e4-4ef1-9ceb-210cbd671a54-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d69cb72-f4e4-4ef1-9ceb-210cbd671a54-credential-socket\") pod \"9d69cb72-f4e4-4ef1-9ceb-210cbd671a54\" (UID: \"9d69cb72-f4e4-4ef1-9ceb-210cbd671a54\") "
Nov 01 06:02:21.009532 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:21.009328    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d69cb72-f4e4-4ef1-9ceb-210cbd671a54-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d69cb72-f4e4-4ef1-9ceb-210cbd671a54-workload-socket\") pod \"9d69cb72-f4e4-4ef1-9ceb-210cbd671a54\" (UID: \"9d69cb72-f4e4-4ef1-9ceb-210cbd671a54\") "
Nov 01 06:02:21.009532 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:21.009395    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d69cb72-f4e4-4ef1-9ceb-210cbd671a54-istio-token\" (UniqueName: \"kubernetes.io/projected/9d69cb72-f4e4-4ef1-9ceb-210cbd671a54-istio-token\") pod \"9d69cb72-f4e4-4ef1-9ceb-210cbd671a54\" (UID: \"9d69cb72-f4e4-4ef1-9ceb-210cbd671a54\") "
Nov 01 06:02:21.009532 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:21.009498    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d69cb72-f4e4-4ef1-9ceb-210cbd671a54-kube-api-access-lgs5b\" (UniqueName: \"kubernetes.io/projected/9d69cb72-f4e4-4ef1-9ceb-210cbd671a54-kube-api-access-lgs5b\") pod \"9d69cb72-f4e4-4ef1-9ceb-210cbd671a54\" (UID: \"9d69cb72-f4e4-4ef1-9ceb-210cbd671a54\") "
Nov 01 06:02:21.009696 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:21.009562    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d69cb72-f4e4-4ef1-9ceb-210cbd671a54-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d69cb72-f4e4-4ef1-9ceb-210cbd671a54-istio-envoy\") pod \"9d69cb72-f4e4-4ef1-9ceb-210cbd671a54\" (UID: \"9d69cb72-f4e4-4ef1-9ceb-210cbd671a54\") "
Nov 01 06:02:21.009696 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:21.009612    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d69cb72-f4e4-4ef1-9ceb-210cbd671a54-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d69cb72-f4e4-4ef1-9ceb-210cbd671a54-istio-data\") pod \"9d69cb72-f4e4-4ef1-9ceb-210cbd671a54\" (UID: \"9d69cb72-f4e4-4ef1-9ceb-210cbd671a54\") "
Nov 01 06:02:21.009796 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:21.009610    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d69cb72-f4e4-4ef1-9ceb-210cbd671a54-workload-certs" pod "9d69cb72-f4e4-4ef1-9ceb-210cbd671a54" (UID: "9d69cb72-f4e4-4ef1-9ceb-210cbd671a54"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:02:21.009796 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:21.009588    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d69cb72-f4e4-4ef1-9ceb-210cbd671a54-credential-socket" pod "9d69cb72-f4e4-4ef1-9ceb-210cbd671a54" (UID: "9d69cb72-f4e4-4ef1-9ceb-210cbd671a54"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:02:21.009796 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:21.009613    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d69cb72-f4e4-4ef1-9ceb-210cbd671a54-istiod-ca-cert" pod "9d69cb72-f4e4-4ef1-9ceb-210cbd671a54" (UID: "9d69cb72-f4e4-4ef1-9ceb-210cbd671a54"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:02:21.009943 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:21.009814    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d69cb72-f4e4-4ef1-9ceb-210cbd671a54-istio-data" pod "9d69cb72-f4e4-4ef1-9ceb-210cbd671a54" (UID: "9d69cb72-f4e4-4ef1-9ceb-210cbd671a54"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:02:21.009943 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:21.009828    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d69cb72-f4e4-4ef1-9ceb-210cbd671a54-workload-socket" pod "9d69cb72-f4e4-4ef1-9ceb-210cbd671a54" (UID: "9d69cb72-f4e4-4ef1-9ceb-210cbd671a54"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:02:21.010088 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:21.010069    5689 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d69cb72-f4e4-4ef1-9ceb-210cbd671a54-credential-socket\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:02:21.010152 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:21.010094    5689 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d69cb72-f4e4-4ef1-9ceb-210cbd671a54-workload-socket\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:02:21.010152 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:21.010110    5689 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d69cb72-f4e4-4ef1-9ceb-210cbd671a54-istio-data\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:02:21.010152 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:21.010125    5689 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d69cb72-f4e4-4ef1-9ceb-210cbd671a54-istiod-ca-cert\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:02:21.010152 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:21.010139    5689 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d69cb72-f4e4-4ef1-9ceb-210cbd671a54-workload-certs\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:02:21.011700 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:21.011670    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d69cb72-f4e4-4ef1-9ceb-210cbd671a54-istio-podinfo" pod "9d69cb72-f4e4-4ef1-9ceb-210cbd671a54" (UID: "9d69cb72-f4e4-4ef1-9ceb-210cbd671a54"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:02:21.011828 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:21.011744    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d69cb72-f4e4-4ef1-9ceb-210cbd671a54-istio-envoy" pod "9d69cb72-f4e4-4ef1-9ceb-210cbd671a54" (UID: "9d69cb72-f4e4-4ef1-9ceb-210cbd671a54"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:02:21.011874 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:21.011824    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d69cb72-f4e4-4ef1-9ceb-210cbd671a54-istio-token" pod "9d69cb72-f4e4-4ef1-9ceb-210cbd671a54" (UID: "9d69cb72-f4e4-4ef1-9ceb-210cbd671a54"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:02:21.012059 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:21.012030    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d69cb72-f4e4-4ef1-9ceb-210cbd671a54-kube-api-access-lgs5b" pod "9d69cb72-f4e4-4ef1-9ceb-210cbd671a54" (UID: "9d69cb72-f4e4-4ef1-9ceb-210cbd671a54"). InnerVolumeSpecName "kube-api-access-lgs5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:02:21.095725 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:21.095678    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:02:21.119496 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:21.119406    5689 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d69cb72-f4e4-4ef1-9ceb-210cbd671a54-istio-podinfo\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:02:21.119496 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:21.119480    5689 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d69cb72-f4e4-4ef1-9ceb-210cbd671a54-istio-token\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:02:21.119738 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:21.119506    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgs5b\" (UniqueName: \"kubernetes.io/projected/9d69cb72-f4e4-4ef1-9ceb-210cbd671a54-kube-api-access-lgs5b\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:02:21.119738 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:21.119527    5689 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d69cb72-f4e4-4ef1-9ceb-210cbd671a54-istio-envoy\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:02:21.681644 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:21.681613    5689 generic.go:356] "Generic (PLEG): container finished" podID="9d69cb72-f4e4-4ef1-9ceb-210cbd671a54" containerID="7c9ff3969a214671a9128620947f7f6691626fcd9fc594c29d3568c4a79e9642" exitCode=0
Nov 01 06:02:21.681918 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:21.681658    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx"
Nov 01 06:02:21.681918 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:21.681676    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx" event={"ID":"9d69cb72-f4e4-4ef1-9ceb-210cbd671a54","Type":"ContainerDied","Data":"7c9ff3969a214671a9128620947f7f6691626fcd9fc594c29d3568c4a79e9642"}
Nov 01 06:02:21.681918 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:21.681718    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx" event={"ID":"9d69cb72-f4e4-4ef1-9ceb-210cbd671a54","Type":"ContainerDied","Data":"2a448cfa95988a97e98f7b7a57dfef345897e7f7ab336e0a0794e11fbe9e3d6c"}
Nov 01 06:02:21.681918 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:21.681738    5689 scope.go:122] "RemoveContainer" containerID="7c9ff3969a214671a9128620947f7f6691626fcd9fc594c29d3568c4a79e9642"
Nov 01 06:02:21.692246 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:21.692227    5689 scope.go:122] "RemoveContainer" containerID="db8bd7e6c8fc393881c66a58e1d3a50e711cf8f9542a92a6752657aac397097b"
Nov 01 06:02:21.700717 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:21.700700    5689 scope.go:122] "RemoveContainer" containerID="7c9ff3969a214671a9128620947f7f6691626fcd9fc594c29d3568c4a79e9642"
Nov 01 06:02:21.700995 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:02:21.700975    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c9ff3969a214671a9128620947f7f6691626fcd9fc594c29d3568c4a79e9642\": container with ID starting with 7c9ff3969a214671a9128620947f7f6691626fcd9fc594c29d3568c4a79e9642 not found: ID does not exist" containerID="7c9ff3969a214671a9128620947f7f6691626fcd9fc594c29d3568c4a79e9642"
Nov 01 06:02:21.701073 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:21.701005    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c9ff3969a214671a9128620947f7f6691626fcd9fc594c29d3568c4a79e9642"} err="failed to get container status \"7c9ff3969a214671a9128620947f7f6691626fcd9fc594c29d3568c4a79e9642\": rpc error: code = NotFound desc = could not find container \"7c9ff3969a214671a9128620947f7f6691626fcd9fc594c29d3568c4a79e9642\": container with ID starting with 7c9ff3969a214671a9128620947f7f6691626fcd9fc594c29d3568c4a79e9642 not found: ID does not exist"
Nov 01 06:02:21.701073 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:21.701029    5689 scope.go:122] "RemoveContainer" containerID="db8bd7e6c8fc393881c66a58e1d3a50e711cf8f9542a92a6752657aac397097b"
Nov 01 06:02:21.701283 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:02:21.701264    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db8bd7e6c8fc393881c66a58e1d3a50e711cf8f9542a92a6752657aac397097b\": container with ID starting with db8bd7e6c8fc393881c66a58e1d3a50e711cf8f9542a92a6752657aac397097b not found: ID does not exist" containerID="db8bd7e6c8fc393881c66a58e1d3a50e711cf8f9542a92a6752657aac397097b"
Nov 01 06:02:21.701324 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:21.701291    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db8bd7e6c8fc393881c66a58e1d3a50e711cf8f9542a92a6752657aac397097b"} err="failed to get container status \"db8bd7e6c8fc393881c66a58e1d3a50e711cf8f9542a92a6752657aac397097b\": rpc error: code = NotFound desc = could not find container \"db8bd7e6c8fc393881c66a58e1d3a50e711cf8f9542a92a6752657aac397097b\": container with ID starting with db8bd7e6c8fc393881c66a58e1d3a50e711cf8f9542a92a6752657aac397097b not found: ID does not exist"
Nov 01 06:02:21.897726 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:21.897694    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx"]
Nov 01 06:02:21.948435 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:21.948362    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-c73135-openshift-default-69c4df7c97-mhljx"]
Nov 01 06:02:22.068287 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:22.068259    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d69cb72-f4e4-4ef1-9ceb-210cbd671a54" path="/var/lib/kubelet/pods/9d69cb72-f4e4-4ef1-9ceb-210cbd671a54/volumes"
Nov 01 06:02:22.636425 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:22.636394    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:02:23.005436 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:23.005394    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-5bms2"]
Nov 01 06:02:23.005962 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:23.005947    5689 cpu_manager.go:414] "containerMap: removing container" podUID="82632280-5edc-47e8-a570-f98de80de625" containerName="istio-proxy"
Nov 01 06:02:23.006006 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:23.005963    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82632280-5edc-47e8-a570-f98de80de625" containerName="istio-proxy"
Nov 01 06:02:23.006006 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:23.005985    5689 cpu_manager.go:414] "containerMap: removing container" podUID="9d69cb72-f4e4-4ef1-9ceb-210cbd671a54" containerName="istio-proxy"
Nov 01 06:02:23.006006 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:23.005990    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d69cb72-f4e4-4ef1-9ceb-210cbd671a54" containerName="istio-proxy"
Nov 01 06:02:23.006006 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:23.006001    5689 cpu_manager.go:414] "containerMap: removing container" podUID="758d79fd-92ca-46fb-b487-694dc9818b61" containerName="istio-proxy"
Nov 01 06:02:23.006006 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:23.006006    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="758d79fd-92ca-46fb-b487-694dc9818b61" containerName="istio-proxy"
Nov 01 06:02:23.006160 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:23.006030    5689 cpu_manager.go:414] "containerMap: removing container" podUID="5b840b67-ec8d-45e0-91b4-0a8dc7a353eb" containerName="istio-proxy"
Nov 01 06:02:23.006160 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:23.006035    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b840b67-ec8d-45e0-91b4-0a8dc7a353eb" containerName="istio-proxy"
Nov 01 06:02:23.006160 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:23.006046    5689 cpu_manager.go:414] "containerMap: removing container" podUID="da1ad4a4-629e-4ce5-a336-1bf832074db4" containerName="istio-proxy"
Nov 01 06:02:23.006160 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:23.006051    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da1ad4a4-629e-4ce5-a336-1bf832074db4" containerName="istio-proxy"
Nov 01 06:02:23.006160 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:23.006118    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="758d79fd-92ca-46fb-b487-694dc9818b61" containerName="istio-proxy"
Nov 01 06:02:23.006160 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:23.006132    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b840b67-ec8d-45e0-91b4-0a8dc7a353eb" containerName="istio-proxy"
Nov 01 06:02:23.006160 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:23.006140    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="da1ad4a4-629e-4ce5-a336-1bf832074db4" containerName="istio-proxy"
Nov 01 06:02:23.006160 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:23.006146    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d69cb72-f4e4-4ef1-9ceb-210cbd671a54" containerName="istio-proxy"
Nov 01 06:02:23.006160 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:23.006152    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="82632280-5edc-47e8-a570-f98de80de625" containerName="istio-proxy"
Nov 01 06:02:23.006160 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:23.006159    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d69cb72-f4e4-4ef1-9ceb-210cbd671a54" containerName="istio-proxy"
Nov 01 06:02:23.010890 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:23.010870    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-5bms2"
Nov 01 06:02:23.114826 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:23.114801    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-5bms2"]
Nov 01 06:02:23.144178 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:23.144153    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea6d76c0-414c-4ae6-9c9a-01e41971592f-workload-socket\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-5bms2\" (UID: \"ea6d76c0-414c-4ae6-9c9a-01e41971592f\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-5bms2"
Nov 01 06:02:23.144339 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:23.144195    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea6d76c0-414c-4ae6-9c9a-01e41971592f-istio-envoy\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-5bms2\" (UID: \"ea6d76c0-414c-4ae6-9c9a-01e41971592f\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-5bms2"
Nov 01 06:02:23.144339 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:23.144221    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea6d76c0-414c-4ae6-9c9a-01e41971592f-istio-data\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-5bms2\" (UID: \"ea6d76c0-414c-4ae6-9c9a-01e41971592f\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-5bms2"
Nov 01 06:02:23.144446 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:23.144350    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea6d76c0-414c-4ae6-9c9a-01e41971592f-istiod-ca-cert\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-5bms2\" (UID: \"ea6d76c0-414c-4ae6-9c9a-01e41971592f\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-5bms2"
Nov 01 06:02:23.144446 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:23.144373    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea6d76c0-414c-4ae6-9c9a-01e41971592f-istio-token\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-5bms2\" (UID: \"ea6d76c0-414c-4ae6-9c9a-01e41971592f\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-5bms2"
Nov 01 06:02:23.144446 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:23.144398    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mpqq\" (UniqueName: \"kubernetes.io/projected/ea6d76c0-414c-4ae6-9c9a-01e41971592f-kube-api-access-6mpqq\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-5bms2\" (UID: \"ea6d76c0-414c-4ae6-9c9a-01e41971592f\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-5bms2"
Nov 01 06:02:23.144446 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:23.144464    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea6d76c0-414c-4ae6-9c9a-01e41971592f-istio-podinfo\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-5bms2\" (UID: \"ea6d76c0-414c-4ae6-9c9a-01e41971592f\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-5bms2"
Nov 01 06:02:23.144668 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:23.144522    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea6d76c0-414c-4ae6-9c9a-01e41971592f-workload-certs\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-5bms2\" (UID: \"ea6d76c0-414c-4ae6-9c9a-01e41971592f\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-5bms2"
Nov 01 06:02:23.144668 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:23.144616    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea6d76c0-414c-4ae6-9c9a-01e41971592f-credential-socket\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-5bms2\" (UID: \"ea6d76c0-414c-4ae6-9c9a-01e41971592f\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-5bms2"
Nov 01 06:02:23.245802 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:23.245774    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea6d76c0-414c-4ae6-9c9a-01e41971592f-workload-socket\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-5bms2\" (UID: \"ea6d76c0-414c-4ae6-9c9a-01e41971592f\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-5bms2"
Nov 01 06:02:23.245952 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:23.245821    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea6d76c0-414c-4ae6-9c9a-01e41971592f-istio-envoy\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-5bms2\" (UID: \"ea6d76c0-414c-4ae6-9c9a-01e41971592f\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-5bms2"
Nov 01 06:02:23.245952 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:23.245851    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea6d76c0-414c-4ae6-9c9a-01e41971592f-istio-data\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-5bms2\" (UID: \"ea6d76c0-414c-4ae6-9c9a-01e41971592f\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-5bms2"
Nov 01 06:02:23.245952 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:23.245909    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea6d76c0-414c-4ae6-9c9a-01e41971592f-istiod-ca-cert\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-5bms2\" (UID: \"ea6d76c0-414c-4ae6-9c9a-01e41971592f\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-5bms2"
Nov 01 06:02:23.245952 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:23.245926    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea6d76c0-414c-4ae6-9c9a-01e41971592f-istio-token\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-5bms2\" (UID: \"ea6d76c0-414c-4ae6-9c9a-01e41971592f\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-5bms2"
Nov 01 06:02:23.246159 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:23.245956    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mpqq\" (UniqueName: \"kubernetes.io/projected/ea6d76c0-414c-4ae6-9c9a-01e41971592f-kube-api-access-6mpqq\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-5bms2\" (UID: \"ea6d76c0-414c-4ae6-9c9a-01e41971592f\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-5bms2"
Nov 01 06:02:23.246159 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:23.245995    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea6d76c0-414c-4ae6-9c9a-01e41971592f-istio-podinfo\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-5bms2\" (UID: \"ea6d76c0-414c-4ae6-9c9a-01e41971592f\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-5bms2"
Nov 01 06:02:23.246159 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:23.246048    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea6d76c0-414c-4ae6-9c9a-01e41971592f-workload-certs\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-5bms2\" (UID: \"ea6d76c0-414c-4ae6-9c9a-01e41971592f\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-5bms2"
Nov 01 06:02:23.246159 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:23.246082    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea6d76c0-414c-4ae6-9c9a-01e41971592f-credential-socket\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-5bms2\" (UID: \"ea6d76c0-414c-4ae6-9c9a-01e41971592f\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-5bms2"
Nov 01 06:02:23.246377 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:23.246213    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea6d76c0-414c-4ae6-9c9a-01e41971592f-workload-socket\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-5bms2\" (UID: \"ea6d76c0-414c-4ae6-9c9a-01e41971592f\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-5bms2"
Nov 01 06:02:23.246377 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:23.246224    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea6d76c0-414c-4ae6-9c9a-01e41971592f-istio-data\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-5bms2\" (UID: \"ea6d76c0-414c-4ae6-9c9a-01e41971592f\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-5bms2"
Nov 01 06:02:23.246562 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:23.246539    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea6d76c0-414c-4ae6-9c9a-01e41971592f-credential-socket\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-5bms2\" (UID: \"ea6d76c0-414c-4ae6-9c9a-01e41971592f\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-5bms2"
Nov 01 06:02:23.246694 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:23.246549    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea6d76c0-414c-4ae6-9c9a-01e41971592f-workload-certs\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-5bms2\" (UID: \"ea6d76c0-414c-4ae6-9c9a-01e41971592f\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-5bms2"
Nov 01 06:02:23.246694 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:23.246639    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea6d76c0-414c-4ae6-9c9a-01e41971592f-istiod-ca-cert\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-5bms2\" (UID: \"ea6d76c0-414c-4ae6-9c9a-01e41971592f\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-5bms2"
Nov 01 06:02:23.248325 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:23.248300    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea6d76c0-414c-4ae6-9c9a-01e41971592f-istio-envoy\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-5bms2\" (UID: \"ea6d76c0-414c-4ae6-9c9a-01e41971592f\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-5bms2"
Nov 01 06:02:23.248577 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:23.248559    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea6d76c0-414c-4ae6-9c9a-01e41971592f-istio-podinfo\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-5bms2\" (UID: \"ea6d76c0-414c-4ae6-9c9a-01e41971592f\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-5bms2"
Nov 01 06:02:23.306438 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:23.306380    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea6d76c0-414c-4ae6-9c9a-01e41971592f-istio-token\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-5bms2\" (UID: \"ea6d76c0-414c-4ae6-9c9a-01e41971592f\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-5bms2"
Nov 01 06:02:23.306537 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:23.306476    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mpqq\" (UniqueName: \"kubernetes.io/projected/ea6d76c0-414c-4ae6-9c9a-01e41971592f-kube-api-access-6mpqq\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-5bms2\" (UID: \"ea6d76c0-414c-4ae6-9c9a-01e41971592f\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-5bms2"
Nov 01 06:02:23.325432 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:23.325408    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-5bms2"
Nov 01 06:02:23.331233 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:23.331215    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-7a360a-6cf998897-7v6d7"]
Nov 01 06:02:23.443738 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:23.443705    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-7a360a-d4b7c96cf-cswr9"]
Nov 01 06:02:23.728700 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:23.728671    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:02:23.774349 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:23.770506    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-5bms2"]
Nov 01 06:02:23.797659 ip-10-0-129-92 kubenswrapper[5689]: W1101 06:02:23.797623    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea6d76c0_414c_4ae6_9c9a_01e41971592f.slice/crio-d2aa0f62f61253e21f11de49ec1c58950967de25c5705606a428312fc74d28c9 WatchSource:0}: Error finding container d2aa0f62f61253e21f11de49ec1c58950967de25c5705606a428312fc74d28c9: Status 404 returned error can't find the container with id d2aa0f62f61253e21f11de49ec1c58950967de25c5705606a428312fc74d28c9
Nov 01 06:02:23.800052 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:23.800000    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 06:02:23.800173 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:23.800103    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 06:02:23.800173 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:23.800147    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 06:02:23.911173 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:23.911145    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:02:24.709527 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:24.709485    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-5bms2" event={"ID":"ea6d76c0-414c-4ae6-9c9a-01e41971592f","Type":"ContainerStarted","Data":"c8321bede367ae90e95bf2919048be4282a992a0198f64bca75abbd5e02669cf"}
Nov 01 06:02:24.709935 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:24.709552    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-5bms2" event={"ID":"ea6d76c0-414c-4ae6-9c9a-01e41971592f","Type":"ContainerStarted","Data":"d2aa0f62f61253e21f11de49ec1c58950967de25c5705606a428312fc74d28c9"}
Nov 01 06:02:24.831419 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:24.831363    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-5bms2" podStartSLOduration=2.831348019 podStartE2EDuration="2.831348019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:02:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:02:24.829286681 +0000 UTC m=+3522.900083414" watchObservedRunningTime="2026-11-01 06:02:24.831348019 +0000 UTC m=+3522.902144739"
Nov 01 06:02:25.326353 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:25.326318    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-5bms2"
Nov 01 06:02:25.327967 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:25.327943    5689 patch_prober.go:29] interesting pod/e2e-worker-w4-a48099-openshift-default-577864759-5bms2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Nov 01 06:02:25.328095 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:25.327990    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-5bms2" podUID="ea6d76c0-414c-4ae6-9c9a-01e41971592f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Nov 01 06:02:26.165109 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:26.165079    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:02:26.326419 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:26.326386    5689 patch_prober.go:29] interesting pod/e2e-worker-w4-a48099-openshift-default-577864759-5bms2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Nov 01 06:02:26.326620 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:26.326446    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-5bms2" podUID="ea6d76c0-414c-4ae6-9c9a-01e41971592f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Nov 01 06:02:27.063242 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:27.063206    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-897c87/sub-test-model-897c87-kserve-6d49ff64df-f27f4"]
Nov 01 06:02:27.063873 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:27.063851    5689 cpu_manager.go:414] "containerMap: removing container" podUID="9d69cb72-f4e4-4ef1-9ceb-210cbd671a54" containerName="istio-proxy"
Nov 01 06:02:27.063873 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:27.063873    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d69cb72-f4e4-4ef1-9ceb-210cbd671a54" containerName="istio-proxy"
Nov 01 06:02:27.067285 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:27.067266    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-897c87/sub-test-model-897c87-kserve-6d49ff64df-f27f4"
Nov 01 06:02:27.132951 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:27.132917    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb7pn\" (UniqueName: \"kubernetes.io/projected/451c733b-f9b7-4bc0-a113-a4e7607bba9f-kube-api-access-sb7pn\") pod \"sub-test-model-897c87-kserve-6d49ff64df-f27f4\" (UID: \"451c733b-f9b7-4bc0-a113-a4e7607bba9f\") " pod="ai-tenant-e2e-shared-a-w1-897c87/sub-test-model-897c87-kserve-6d49ff64df-f27f4"
Nov 01 06:02:27.133106 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:27.133056    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/451c733b-f9b7-4bc0-a113-a4e7607bba9f-home\") pod \"sub-test-model-897c87-kserve-6d49ff64df-f27f4\" (UID: \"451c733b-f9b7-4bc0-a113-a4e7607bba9f\") " pod="ai-tenant-e2e-shared-a-w1-897c87/sub-test-model-897c87-kserve-6d49ff64df-f27f4"
Nov 01 06:02:27.133106 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:27.133091    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/451c733b-f9b7-4bc0-a113-a4e7607bba9f-dshm\") pod \"sub-test-model-897c87-kserve-6d49ff64df-f27f4\" (UID: \"451c733b-f9b7-4bc0-a113-a4e7607bba9f\") " pod="ai-tenant-e2e-shared-a-w1-897c87/sub-test-model-897c87-kserve-6d49ff64df-f27f4"
Nov 01 06:02:27.133193 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:27.133174    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/451c733b-f9b7-4bc0-a113-a4e7607bba9f-tls-certs\") pod \"sub-test-model-897c87-kserve-6d49ff64df-f27f4\" (UID: \"451c733b-f9b7-4bc0-a113-a4e7607bba9f\") " pod="ai-tenant-e2e-shared-a-w1-897c87/sub-test-model-897c87-kserve-6d49ff64df-f27f4"
Nov 01 06:02:27.133230 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:27.133209    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/451c733b-f9b7-4bc0-a113-a4e7607bba9f-model-cache\") pod \"sub-test-model-897c87-kserve-6d49ff64df-f27f4\" (UID: \"451c733b-f9b7-4bc0-a113-a4e7607bba9f\") " pod="ai-tenant-e2e-shared-a-w1-897c87/sub-test-model-897c87-kserve-6d49ff64df-f27f4"
Nov 01 06:02:27.133314 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:27.133303    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/451c733b-f9b7-4bc0-a113-a4e7607bba9f-tmp-dir\") pod \"sub-test-model-897c87-kserve-6d49ff64df-f27f4\" (UID: \"451c733b-f9b7-4bc0-a113-a4e7607bba9f\") " pod="ai-tenant-e2e-shared-a-w1-897c87/sub-test-model-897c87-kserve-6d49ff64df-f27f4"
Nov 01 06:02:27.168418 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:27.168393    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-897c87\"/\"openshift-service-ca.crt\""
Nov 01 06:02:27.168418 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:27.168417    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-897c87\"/\"kube-root-ca.crt\""
Nov 01 06:02:27.168904 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:27.168509    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-897c87\"/\"default-dockercfg-xd89h\""
Nov 01 06:02:27.168904 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:27.168548    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-897c87\"/\"sub-test-model-897c87-kserve-self-signed-certs\""
Nov 01 06:02:27.170867 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:27.170845    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-897c87/sub-test-model-897c87-kserve-6d49ff64df-f27f4"]
Nov 01 06:02:27.199612 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:27.198937    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:02:27.234965 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:27.234921    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/451c733b-f9b7-4bc0-a113-a4e7607bba9f-tmp-dir\") pod \"sub-test-model-897c87-kserve-6d49ff64df-f27f4\" (UID: \"451c733b-f9b7-4bc0-a113-a4e7607bba9f\") " pod="ai-tenant-e2e-shared-a-w1-897c87/sub-test-model-897c87-kserve-6d49ff64df-f27f4"
Nov 01 06:02:27.234965 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:27.235048    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb7pn\" (UniqueName: \"kubernetes.io/projected/451c733b-f9b7-4bc0-a113-a4e7607bba9f-kube-api-access-sb7pn\") pod \"sub-test-model-897c87-kserve-6d49ff64df-f27f4\" (UID: \"451c733b-f9b7-4bc0-a113-a4e7607bba9f\") " pod="ai-tenant-e2e-shared-a-w1-897c87/sub-test-model-897c87-kserve-6d49ff64df-f27f4"
Nov 01 06:02:27.235363 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:27.235125    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/451c733b-f9b7-4bc0-a113-a4e7607bba9f-home\") pod \"sub-test-model-897c87-kserve-6d49ff64df-f27f4\" (UID: \"451c733b-f9b7-4bc0-a113-a4e7607bba9f\") " pod="ai-tenant-e2e-shared-a-w1-897c87/sub-test-model-897c87-kserve-6d49ff64df-f27f4"
Nov 01 06:02:27.235363 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:27.235156    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/451c733b-f9b7-4bc0-a113-a4e7607bba9f-dshm\") pod \"sub-test-model-897c87-kserve-6d49ff64df-f27f4\" (UID: \"451c733b-f9b7-4bc0-a113-a4e7607bba9f\") " pod="ai-tenant-e2e-shared-a-w1-897c87/sub-test-model-897c87-kserve-6d49ff64df-f27f4"
Nov 01 06:02:27.235363 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:27.235200    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/451c733b-f9b7-4bc0-a113-a4e7607bba9f-tls-certs\") pod \"sub-test-model-897c87-kserve-6d49ff64df-f27f4\" (UID: \"451c733b-f9b7-4bc0-a113-a4e7607bba9f\") " pod="ai-tenant-e2e-shared-a-w1-897c87/sub-test-model-897c87-kserve-6d49ff64df-f27f4"
Nov 01 06:02:27.235363 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:27.235234    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/451c733b-f9b7-4bc0-a113-a4e7607bba9f-model-cache\") pod \"sub-test-model-897c87-kserve-6d49ff64df-f27f4\" (UID: \"451c733b-f9b7-4bc0-a113-a4e7607bba9f\") " pod="ai-tenant-e2e-shared-a-w1-897c87/sub-test-model-897c87-kserve-6d49ff64df-f27f4"
Nov 01 06:02:27.235859 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:27.235447    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/451c733b-f9b7-4bc0-a113-a4e7607bba9f-tmp-dir\") pod \"sub-test-model-897c87-kserve-6d49ff64df-f27f4\" (UID: \"451c733b-f9b7-4bc0-a113-a4e7607bba9f\") " pod="ai-tenant-e2e-shared-a-w1-897c87/sub-test-model-897c87-kserve-6d49ff64df-f27f4"
Nov 01 06:02:27.235859 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:27.235636    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/451c733b-f9b7-4bc0-a113-a4e7607bba9f-model-cache\") pod \"sub-test-model-897c87-kserve-6d49ff64df-f27f4\" (UID: \"451c733b-f9b7-4bc0-a113-a4e7607bba9f\") " pod="ai-tenant-e2e-shared-a-w1-897c87/sub-test-model-897c87-kserve-6d49ff64df-f27f4"
Nov 01 06:02:27.237058 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:27.237035    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/451c733b-f9b7-4bc0-a113-a4e7607bba9f-home\") pod \"sub-test-model-897c87-kserve-6d49ff64df-f27f4\" (UID: \"451c733b-f9b7-4bc0-a113-a4e7607bba9f\") " pod="ai-tenant-e2e-shared-a-w1-897c87/sub-test-model-897c87-kserve-6d49ff64df-f27f4"
Nov 01 06:02:27.239139 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:27.239119    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/451c733b-f9b7-4bc0-a113-a4e7607bba9f-dshm\") pod \"sub-test-model-897c87-kserve-6d49ff64df-f27f4\" (UID: \"451c733b-f9b7-4bc0-a113-a4e7607bba9f\") " pod="ai-tenant-e2e-shared-a-w1-897c87/sub-test-model-897c87-kserve-6d49ff64df-f27f4"
Nov 01 06:02:27.241958 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:27.241935    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/451c733b-f9b7-4bc0-a113-a4e7607bba9f-tls-certs\") pod \"sub-test-model-897c87-kserve-6d49ff64df-f27f4\" (UID: \"451c733b-f9b7-4bc0-a113-a4e7607bba9f\") " pod="ai-tenant-e2e-shared-a-w1-897c87/sub-test-model-897c87-kserve-6d49ff64df-f27f4"
Nov 01 06:02:27.300951 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:27.300921    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb7pn\" (UniqueName: \"kubernetes.io/projected/451c733b-f9b7-4bc0-a113-a4e7607bba9f-kube-api-access-sb7pn\") pod \"sub-test-model-897c87-kserve-6d49ff64df-f27f4\" (UID: \"451c733b-f9b7-4bc0-a113-a4e7607bba9f\") " pod="ai-tenant-e2e-shared-a-w1-897c87/sub-test-model-897c87-kserve-6d49ff64df-f27f4"
Nov 01 06:02:27.325911 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:27.325839    5689 patch_prober.go:29] interesting pod/e2e-worker-w4-a48099-openshift-default-577864759-5bms2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Nov 01 06:02:27.326056 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:27.325910    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-5bms2" podUID="ea6d76c0-414c-4ae6-9c9a-01e41971592f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Nov 01 06:02:27.408317 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:27.408274    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-897c87/sub-test-model-897c87-kserve-6d49ff64df-f27f4"
Nov 01 06:02:27.800034 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:27.799987    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-897c87/sub-test-model-897c87-kserve-6d49ff64df-f27f4"]
Nov 01 06:02:27.804902 ip-10-0-129-92 kubenswrapper[5689]: W1101 06:02:27.804879    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod451c733b_f9b7_4bc0_a113_a4e7607bba9f.slice/crio-b0d596388662c6d1b1e2049ad0daea9fee8859cb37daddaf1f1c3a5af030fb9c WatchSource:0}: Error finding container b0d596388662c6d1b1e2049ad0daea9fee8859cb37daddaf1f1c3a5af030fb9c: Status 404 returned error can't find the container with id b0d596388662c6d1b1e2049ad0daea9fee8859cb37daddaf1f1c3a5af030fb9c
Nov 01 06:02:28.326198 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:28.326162    5689 patch_prober.go:29] interesting pod/e2e-worker-w4-a48099-openshift-default-577864759-5bms2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Nov 01 06:02:28.326584 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:28.326242    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-5bms2" podUID="ea6d76c0-414c-4ae6-9c9a-01e41971592f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Nov 01 06:02:28.332360 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:28.332331    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-7a360a-6cf998897-7v6d7" podUID="838940c1-785d-4fb4-9c0a-c967b252ff64" containerName="payload-processing" containerID="cri-o://1dcc327fee13764dfbf2d2ea79754c68b9ee33654115411bc0791e444b04846b" gracePeriod=25
Nov 01 06:02:28.445098 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:28.444526    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7a360a-d4b7c96cf-cswr9" podUID="1177a649-2110-4dae-86a9-dc801392d954" containerName="payload-pre-processing" containerID="cri-o://ab76d949771bfee429a81a4f4622e1e3551be494f932402b0f9f0d3725bfd002" gracePeriod=25
Nov 01 06:02:28.637631 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:28.637603    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-7a360a-6cf998897-7v6d7"
Nov 01 06:02:28.662543 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:28.661136    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/838940c1-785d-4fb4-9c0a-c967b252ff64-kube-api-access-gt2h8\" (UniqueName: \"kubernetes.io/projected/838940c1-785d-4fb4-9c0a-c967b252ff64-kube-api-access-gt2h8\") pod \"838940c1-785d-4fb4-9c0a-c967b252ff64\" (UID: \"838940c1-785d-4fb4-9c0a-c967b252ff64\") "
Nov 01 06:02:28.662543 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:28.661168    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/838940c1-785d-4fb4-9c0a-c967b252ff64-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/838940c1-785d-4fb4-9c0a-c967b252ff64-plugins-config-volume\") pod \"838940c1-785d-4fb4-9c0a-c967b252ff64\" (UID: \"838940c1-785d-4fb4-9c0a-c967b252ff64\") "
Nov 01 06:02:28.662543 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:28.661911    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/838940c1-785d-4fb4-9c0a-c967b252ff64-plugins-config-volume" pod "838940c1-785d-4fb4-9c0a-c967b252ff64" (UID: "838940c1-785d-4fb4-9c0a-c967b252ff64"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:02:28.664255 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:28.664229    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/838940c1-785d-4fb4-9c0a-c967b252ff64-kube-api-access-gt2h8" pod "838940c1-785d-4fb4-9c0a-c967b252ff64" (UID: "838940c1-785d-4fb4-9c0a-c967b252ff64"). InnerVolumeSpecName "kube-api-access-gt2h8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:02:28.732433 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:28.732397    5689 generic.go:356] "Generic (PLEG): container finished" podID="1177a649-2110-4dae-86a9-dc801392d954" containerID="ab76d949771bfee429a81a4f4622e1e3551be494f932402b0f9f0d3725bfd002" exitCode=0
Nov 01 06:02:28.732615 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:28.732542    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7a360a-d4b7c96cf-cswr9" event={"ID":"1177a649-2110-4dae-86a9-dc801392d954","Type":"ContainerDied","Data":"ab76d949771bfee429a81a4f4622e1e3551be494f932402b0f9f0d3725bfd002"}
Nov 01 06:02:28.734100 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:28.734079    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-897c87/sub-test-model-897c87-kserve-6d49ff64df-f27f4" event={"ID":"451c733b-f9b7-4bc0-a113-a4e7607bba9f","Type":"ContainerStarted","Data":"a0a4cd26fb3fe18f2c2f6cd114f6ce5f5dbf5c4429bf58ed367fbebbbbd84301"}
Nov 01 06:02:28.734679 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:28.734661    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-897c87/sub-test-model-897c87-kserve-6d49ff64df-f27f4" event={"ID":"451c733b-f9b7-4bc0-a113-a4e7607bba9f","Type":"ContainerStarted","Data":"b0d596388662c6d1b1e2049ad0daea9fee8859cb37daddaf1f1c3a5af030fb9c"}
Nov 01 06:02:28.735735 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:28.735710    5689 generic.go:356] "Generic (PLEG): container finished" podID="838940c1-785d-4fb4-9c0a-c967b252ff64" containerID="1dcc327fee13764dfbf2d2ea79754c68b9ee33654115411bc0791e444b04846b" exitCode=0
Nov 01 06:02:28.735830 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:28.735778    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-7a360a-6cf998897-7v6d7" event={"ID":"838940c1-785d-4fb4-9c0a-c967b252ff64","Type":"ContainerDied","Data":"1dcc327fee13764dfbf2d2ea79754c68b9ee33654115411bc0791e444b04846b"}
Nov 01 06:02:28.735830 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:28.735815    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-7a360a-6cf998897-7v6d7" event={"ID":"838940c1-785d-4fb4-9c0a-c967b252ff64","Type":"ContainerDied","Data":"81849d357325aee59495b078d29e225047f54fd96efe5cbd6f3c1d6798b25ac9"}
Nov 01 06:02:28.735900 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:28.735844    5689 scope.go:122] "RemoveContainer" containerID="1dcc327fee13764dfbf2d2ea79754c68b9ee33654115411bc0791e444b04846b"
Nov 01 06:02:28.736002 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:28.735988    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-7a360a-6cf998897-7v6d7"
Nov 01 06:02:28.747010 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:28.746943    5689 scope.go:122] "RemoveContainer" containerID="1dcc327fee13764dfbf2d2ea79754c68b9ee33654115411bc0791e444b04846b"
Nov 01 06:02:28.747237 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:02:28.747214    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dcc327fee13764dfbf2d2ea79754c68b9ee33654115411bc0791e444b04846b\": container with ID starting with 1dcc327fee13764dfbf2d2ea79754c68b9ee33654115411bc0791e444b04846b not found: ID does not exist" containerID="1dcc327fee13764dfbf2d2ea79754c68b9ee33654115411bc0791e444b04846b"
Nov 01 06:02:28.747311 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:28.747249    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dcc327fee13764dfbf2d2ea79754c68b9ee33654115411bc0791e444b04846b"} err="failed to get container status \"1dcc327fee13764dfbf2d2ea79754c68b9ee33654115411bc0791e444b04846b\": rpc error: code = NotFound desc = could not find container \"1dcc327fee13764dfbf2d2ea79754c68b9ee33654115411bc0791e444b04846b\": container with ID starting with 1dcc327fee13764dfbf2d2ea79754c68b9ee33654115411bc0791e444b04846b not found: ID does not exist"
Nov 01 06:02:28.750921 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:28.750902    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7a360a-d4b7c96cf-cswr9"
Nov 01 06:02:28.764027 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:28.764001    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gt2h8\" (UniqueName: \"kubernetes.io/projected/838940c1-785d-4fb4-9c0a-c967b252ff64-kube-api-access-gt2h8\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:02:28.764027 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:28.764021    5689 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/838940c1-785d-4fb4-9c0a-c967b252ff64-plugins-config-volume\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:02:28.864947 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:28.864872    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1177a649-2110-4dae-86a9-dc801392d954-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1177a649-2110-4dae-86a9-dc801392d954-plugins-config-volume\") pod \"1177a649-2110-4dae-86a9-dc801392d954\" (UID: \"1177a649-2110-4dae-86a9-dc801392d954\") "
Nov 01 06:02:28.864947 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:28.864912    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1177a649-2110-4dae-86a9-dc801392d954-kube-api-access-qcwgm\" (UniqueName: \"kubernetes.io/projected/1177a649-2110-4dae-86a9-dc801392d954-kube-api-access-qcwgm\") pod \"1177a649-2110-4dae-86a9-dc801392d954\" (UID: \"1177a649-2110-4dae-86a9-dc801392d954\") "
Nov 01 06:02:28.865312 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:28.865287    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1177a649-2110-4dae-86a9-dc801392d954-plugins-config-volume" pod "1177a649-2110-4dae-86a9-dc801392d954" (UID: "1177a649-2110-4dae-86a9-dc801392d954"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:02:28.867130 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:28.867110    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1177a649-2110-4dae-86a9-dc801392d954-kube-api-access-qcwgm" pod "1177a649-2110-4dae-86a9-dc801392d954" (UID: "1177a649-2110-4dae-86a9-dc801392d954"). InnerVolumeSpecName "kube-api-access-qcwgm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:02:28.966418 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:28.966386    5689 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1177a649-2110-4dae-86a9-dc801392d954-plugins-config-volume\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:02:28.966418 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:28.966411    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcwgm\" (UniqueName: \"kubernetes.io/projected/1177a649-2110-4dae-86a9-dc801392d954-kube-api-access-qcwgm\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:02:29.003255 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:29.003200    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-897c87/sub-test-model-897c87-kserve-6d49ff64df-f27f4" podStartSLOduration=2.003185171 podStartE2EDuration="2.003185171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:02:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:02:29.001219412 +0000 UTC m=+3527.072016133" watchObservedRunningTime="2026-11-01 06:02:29.003185171 +0000 UTC m=+3527.073981891"
Nov 01 06:02:29.121799 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:29.115310    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-7a360a-6cf998897-7v6d7"]
Nov 01 06:02:29.121799 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:29.119180    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-7a360a-6cf998897-7v6d7"]
Nov 01 06:02:29.325908 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:29.325871    5689 patch_prober.go:29] interesting pod/e2e-worker-w4-a48099-openshift-default-577864759-5bms2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Nov 01 06:02:29.326070 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:29.325932    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-5bms2" podUID="ea6d76c0-414c-4ae6-9c9a-01e41971592f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Nov 01 06:02:29.351000 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:29.350971    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:02:29.408710 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:29.408684    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-897c87/sub-test-model-897c87-kserve-6d49ff64df-f27f4"
Nov 01 06:02:29.743681 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:29.743602    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7a360a-d4b7c96cf-cswr9"
Nov 01 06:02:29.743681 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:29.743639    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7a360a-d4b7c96cf-cswr9" event={"ID":"1177a649-2110-4dae-86a9-dc801392d954","Type":"ContainerDied","Data":"bfa908e32db75eef7e600ce051fdd42a976b60ebc4fec6749173d67383d58896"}
Nov 01 06:02:29.743887 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:29.743693    5689 scope.go:122] "RemoveContainer" containerID="ab76d949771bfee429a81a4f4622e1e3551be494f932402b0f9f0d3725bfd002"
Nov 01 06:02:29.906931 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:29.906893    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-7a360a-d4b7c96cf-cswr9"]
Nov 01 06:02:29.909073 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:29.909024    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-7a360a-d4b7c96cf-cswr9"]
Nov 01 06:02:30.068429 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:30.068351    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1177a649-2110-4dae-86a9-dc801392d954" path="/var/lib/kubelet/pods/1177a649-2110-4dae-86a9-dc801392d954/volumes"
Nov 01 06:02:30.068725 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:02:30.068698    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 06:02:30.068827 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:30.068812    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="838940c1-785d-4fb4-9c0a-c967b252ff64" path="/var/lib/kubelet/pods/838940c1-785d-4fb4-9c0a-c967b252ff64/volumes"
Nov 01 06:02:30.213854 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:30.213826    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d96a66-7f878b74fd-m8zrw"]
Nov 01 06:02:30.214039 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:30.214019    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d96a66-7f878b74fd-m8zrw" podUID="ef1eafdc-0b49-42ed-ac95-6bddfd5486ec" containerName="maas-api" containerID="cri-o://59557d026bc8d3faa1682b2f60b8990872b98cf1f2c18db6215327fd4336c505" gracePeriod=30
Nov 01 06:02:30.326109 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:30.326039    5689 patch_prober.go:29] interesting pod/e2e-worker-w4-a48099-openshift-default-577864759-5bms2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Nov 01 06:02:30.326109 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:30.326090    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-5bms2" podUID="ea6d76c0-414c-4ae6-9c9a-01e41971592f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Nov 01 06:02:30.339930 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:30.339901    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:02:30.504333 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:30.504306    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d96a66-7f878b74fd-m8zrw"
Nov 01 06:02:30.607527 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:30.586840    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef1eafdc-0b49-42ed-ac95-6bddfd5486ec-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef1eafdc-0b49-42ed-ac95-6bddfd5486ec-openshift-service-ca\") pod \"ef1eafdc-0b49-42ed-ac95-6bddfd5486ec\" (UID: \"ef1eafdc-0b49-42ed-ac95-6bddfd5486ec\") "
Nov 01 06:02:30.607527 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:30.586898    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef1eafdc-0b49-42ed-ac95-6bddfd5486ec-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef1eafdc-0b49-42ed-ac95-6bddfd5486ec-metrics-certs\") pod \"ef1eafdc-0b49-42ed-ac95-6bddfd5486ec\" (UID: \"ef1eafdc-0b49-42ed-ac95-6bddfd5486ec\") "
Nov 01 06:02:30.607527 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:30.587140    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef1eafdc-0b49-42ed-ac95-6bddfd5486ec-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef1eafdc-0b49-42ed-ac95-6bddfd5486ec-maas-api-tls\") pod \"ef1eafdc-0b49-42ed-ac95-6bddfd5486ec\" (UID: \"ef1eafdc-0b49-42ed-ac95-6bddfd5486ec\") "
Nov 01 06:02:30.607527 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:30.587174    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef1eafdc-0b49-42ed-ac95-6bddfd5486ec-kube-api-access-ff8bt\" (UniqueName: \"kubernetes.io/projected/ef1eafdc-0b49-42ed-ac95-6bddfd5486ec-kube-api-access-ff8bt\") pod \"ef1eafdc-0b49-42ed-ac95-6bddfd5486ec\" (UID: \"ef1eafdc-0b49-42ed-ac95-6bddfd5486ec\") "
Nov 01 06:02:30.607527 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:30.588329    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef1eafdc-0b49-42ed-ac95-6bddfd5486ec-openshift-service-ca" pod "ef1eafdc-0b49-42ed-ac95-6bddfd5486ec" (UID: "ef1eafdc-0b49-42ed-ac95-6bddfd5486ec"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:02:30.612482 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:30.611067    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef1eafdc-0b49-42ed-ac95-6bddfd5486ec-kube-api-access-ff8bt" pod "ef1eafdc-0b49-42ed-ac95-6bddfd5486ec" (UID: "ef1eafdc-0b49-42ed-ac95-6bddfd5486ec"). InnerVolumeSpecName "kube-api-access-ff8bt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:02:30.640715 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:30.637255    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef1eafdc-0b49-42ed-ac95-6bddfd5486ec-maas-api-tls" pod "ef1eafdc-0b49-42ed-ac95-6bddfd5486ec" (UID: "ef1eafdc-0b49-42ed-ac95-6bddfd5486ec"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:02:30.640715 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:30.637963    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef1eafdc-0b49-42ed-ac95-6bddfd5486ec-metrics-certs" pod "ef1eafdc-0b49-42ed-ac95-6bddfd5486ec" (UID: "ef1eafdc-0b49-42ed-ac95-6bddfd5486ec"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:02:30.688326 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:30.688297    5689 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef1eafdc-0b49-42ed-ac95-6bddfd5486ec-maas-api-tls\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:02:30.688326 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:30.688326    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ff8bt\" (UniqueName: \"kubernetes.io/projected/ef1eafdc-0b49-42ed-ac95-6bddfd5486ec-kube-api-access-ff8bt\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:02:30.688560 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:30.688337    5689 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef1eafdc-0b49-42ed-ac95-6bddfd5486ec-openshift-service-ca\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:02:30.688560 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:30.688346    5689 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef1eafdc-0b49-42ed-ac95-6bddfd5486ec-metrics-certs\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:02:30.770823 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:30.770786    5689 generic.go:356] "Generic (PLEG): container finished" podID="ef1eafdc-0b49-42ed-ac95-6bddfd5486ec" containerID="59557d026bc8d3faa1682b2f60b8990872b98cf1f2c18db6215327fd4336c505" exitCode=0
Nov 01 06:02:30.771015 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:30.770857    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d96a66-7f878b74fd-m8zrw"
Nov 01 06:02:30.771015 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:30.770880    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d96a66-7f878b74fd-m8zrw" event={"ID":"ef1eafdc-0b49-42ed-ac95-6bddfd5486ec","Type":"ContainerDied","Data":"59557d026bc8d3faa1682b2f60b8990872b98cf1f2c18db6215327fd4336c505"}
Nov 01 06:02:30.771015 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:30.770951    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d96a66-7f878b74fd-m8zrw" event={"ID":"ef1eafdc-0b49-42ed-ac95-6bddfd5486ec","Type":"ContainerDied","Data":"233b55f52e96ca4241349e34d337ffac2a37ac2e5b02f2bcb9d1719848dcdb71"}
Nov 01 06:02:30.771015 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:30.770985    5689 scope.go:122] "RemoveContainer" containerID="59557d026bc8d3faa1682b2f60b8990872b98cf1f2c18db6215327fd4336c505"
Nov 01 06:02:30.781940 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:30.781918    5689 scope.go:122] "RemoveContainer" containerID="59557d026bc8d3faa1682b2f60b8990872b98cf1f2c18db6215327fd4336c505"
Nov 01 06:02:30.783239 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:02:30.783214    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59557d026bc8d3faa1682b2f60b8990872b98cf1f2c18db6215327fd4336c505\": container with ID starting with 59557d026bc8d3faa1682b2f60b8990872b98cf1f2c18db6215327fd4336c505 not found: ID does not exist" containerID="59557d026bc8d3faa1682b2f60b8990872b98cf1f2c18db6215327fd4336c505"
Nov 01 06:02:30.783337 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:30.783250    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59557d026bc8d3faa1682b2f60b8990872b98cf1f2c18db6215327fd4336c505"} err="failed to get container status \"59557d026bc8d3faa1682b2f60b8990872b98cf1f2c18db6215327fd4336c505\": rpc error: code = NotFound desc = could not find container \"59557d026bc8d3faa1682b2f60b8990872b98cf1f2c18db6215327fd4336c505\": container with ID starting with 59557d026bc8d3faa1682b2f60b8990872b98cf1f2c18db6215327fd4336c505 not found: ID does not exist"
Nov 01 06:02:30.935922 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:30.935887    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d96a66-7f878b74fd-m8zrw"]
Nov 01 06:02:30.941522 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:30.941496    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d96a66-7f878b74fd-m8zrw"]
Nov 01 06:02:31.330844 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:31.330770    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-5bms2"
Nov 01 06:02:31.331229 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:31.331217    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-5bms2"
Nov 01 06:02:31.332012 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:31.331996    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-5bms2"
Nov 01 06:02:32.067927 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:32.067892    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef1eafdc-0b49-42ed-ac95-6bddfd5486ec" path="/var/lib/kubelet/pods/ef1eafdc-0b49-42ed-ac95-6bddfd5486ec/volumes"
Nov 01 06:02:32.158295 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:32.158255    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:02:36.908379 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:36.908347    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:02:37.409016 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:37.408982    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-897c87/sub-test-model-897c87-kserve-6d49ff64df-f27f4"
Nov 01 06:02:37.421499 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:37.421474    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-897c87/sub-test-model-897c87-kserve-6d49ff64df-f27f4"
Nov 01 06:02:37.821547 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:37.821473    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-897c87/sub-test-model-897c87-kserve-6d49ff64df-f27f4"
Nov 01 06:02:37.993524 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:37.993479    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:02:38.171914 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:38.171882    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:02:39.611661 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:39.611628    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:02:40.102558 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:40.102516    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:02:40.899072 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:40.899039    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:02:41.348094 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:41.348018    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:02:42.036627 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:42.036592    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:02:46.710635 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:46.710600    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:02:47.105854 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:47.105766    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:02:47.998112 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:47.998080    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:02:48.170124 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:48.170079    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:02:50.463986 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:50.463951    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:02:51.739227 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:51.739188    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:02:52.706957 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:52.706931    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:02:53.542431 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:53.542393    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:02:57.663534 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:57.663502    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:02:57.845273 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:57.845234    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:02:58.700970 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:58.700941    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:02:58.877608 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:58.877573    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:02:59.131332 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:59.131256    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:02:59.646293 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:02:59.646250    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:00.068149 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:03:00.068113    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 06:03:00.429446 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:00.429409    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:03.899665 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:03.899632    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:06.986502 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:06.986468    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-c6e99c-8b7bf895f-hj225"]
Nov 01 06:03:07.003174 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:07.003145    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:07.092625 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:07.092597    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-c6e99c-fcb666fb7-kc9cw"]
Nov 01 06:03:07.384132 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:07.384052    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:07.557230 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:07.557175    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:07.627282 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:07.627251    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:03:08.153375 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:08.153343    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-c6e99c-bbf76fcb8-bbrnt"]
Nov 01 06:03:08.153790 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:08.153584    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c6e99c-bbf76fcb8-bbrnt" podUID="9729d894-945b-4d8b-8700-46d3552c2c8e" containerName="maas-api" containerID="cri-o://788ea62d7ba6f99ef9e2900c01c9a1bdb67362c0b4ffc768bb4643dfb5efd0a5" gracePeriod=30
Nov 01 06:03:08.280832 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:08.280801    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:08.462378 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:08.462294    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c6e99c-bbf76fcb8-bbrnt"
Nov 01 06:03:08.530361 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:08.530333    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9729d894-945b-4d8b-8700-46d3552c2c8e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9729d894-945b-4d8b-8700-46d3552c2c8e-openshift-service-ca\") pod \"9729d894-945b-4d8b-8700-46d3552c2c8e\" (UID: \"9729d894-945b-4d8b-8700-46d3552c2c8e\") "
Nov 01 06:03:08.530555 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:08.530418    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9729d894-945b-4d8b-8700-46d3552c2c8e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9729d894-945b-4d8b-8700-46d3552c2c8e-metrics-certs\") pod \"9729d894-945b-4d8b-8700-46d3552c2c8e\" (UID: \"9729d894-945b-4d8b-8700-46d3552c2c8e\") "
Nov 01 06:03:08.530555 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:08.530491    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9729d894-945b-4d8b-8700-46d3552c2c8e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9729d894-945b-4d8b-8700-46d3552c2c8e-maas-api-tls\") pod \"9729d894-945b-4d8b-8700-46d3552c2c8e\" (UID: \"9729d894-945b-4d8b-8700-46d3552c2c8e\") "
Nov 01 06:03:08.530555 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:08.530540    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9729d894-945b-4d8b-8700-46d3552c2c8e-kube-api-access-6fwp7\" (UniqueName: \"kubernetes.io/projected/9729d894-945b-4d8b-8700-46d3552c2c8e-kube-api-access-6fwp7\") pod \"9729d894-945b-4d8b-8700-46d3552c2c8e\" (UID: \"9729d894-945b-4d8b-8700-46d3552c2c8e\") "
Nov 01 06:03:08.530788 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:08.530761    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9729d894-945b-4d8b-8700-46d3552c2c8e-openshift-service-ca" pod "9729d894-945b-4d8b-8700-46d3552c2c8e" (UID: "9729d894-945b-4d8b-8700-46d3552c2c8e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:03:08.531014 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:08.530989    5689 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9729d894-945b-4d8b-8700-46d3552c2c8e-openshift-service-ca\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:03:08.532577 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:08.532557    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9729d894-945b-4d8b-8700-46d3552c2c8e-metrics-certs" pod "9729d894-945b-4d8b-8700-46d3552c2c8e" (UID: "9729d894-945b-4d8b-8700-46d3552c2c8e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:03:08.532989 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:08.532955    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9729d894-945b-4d8b-8700-46d3552c2c8e-kube-api-access-6fwp7" pod "9729d894-945b-4d8b-8700-46d3552c2c8e" (UID: "9729d894-945b-4d8b-8700-46d3552c2c8e"). InnerVolumeSpecName "kube-api-access-6fwp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:03:08.533861 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:08.533836    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9729d894-945b-4d8b-8700-46d3552c2c8e-maas-api-tls" pod "9729d894-945b-4d8b-8700-46d3552c2c8e" (UID: "9729d894-945b-4d8b-8700-46d3552c2c8e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:03:08.632173 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:08.632135    5689 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9729d894-945b-4d8b-8700-46d3552c2c8e-metrics-certs\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:03:08.632173 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:08.632163    5689 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9729d894-945b-4d8b-8700-46d3552c2c8e-maas-api-tls\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:03:08.632173 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:08.632177    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fwp7\" (UniqueName: \"kubernetes.io/projected/9729d894-945b-4d8b-8700-46d3552c2c8e-kube-api-access-6fwp7\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:03:08.979884 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:08.979853    5689 generic.go:356] "Generic (PLEG): container finished" podID="9729d894-945b-4d8b-8700-46d3552c2c8e" containerID="788ea62d7ba6f99ef9e2900c01c9a1bdb67362c0b4ffc768bb4643dfb5efd0a5" exitCode=0
Nov 01 06:03:08.980058 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:08.979944    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c6e99c-bbf76fcb8-bbrnt" event={"ID":"9729d894-945b-4d8b-8700-46d3552c2c8e","Type":"ContainerDied","Data":"788ea62d7ba6f99ef9e2900c01c9a1bdb67362c0b4ffc768bb4643dfb5efd0a5"}
Nov 01 06:03:08.980058 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:08.979964    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c6e99c-bbf76fcb8-bbrnt"
Nov 01 06:03:08.980058 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:08.979979    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c6e99c-bbf76fcb8-bbrnt" event={"ID":"9729d894-945b-4d8b-8700-46d3552c2c8e","Type":"ContainerDied","Data":"2d006b3081dfcf86a2dedddee85c543d9b0ea32aa8bec24b379354cbf6412917"}
Nov 01 06:03:08.980058 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:08.979999    5689 scope.go:122] "RemoveContainer" containerID="788ea62d7ba6f99ef9e2900c01c9a1bdb67362c0b4ffc768bb4643dfb5efd0a5"
Nov 01 06:03:08.990033 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:08.990012    5689 scope.go:122] "RemoveContainer" containerID="788ea62d7ba6f99ef9e2900c01c9a1bdb67362c0b4ffc768bb4643dfb5efd0a5"
Nov 01 06:03:08.990321 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:03:08.990297    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"788ea62d7ba6f99ef9e2900c01c9a1bdb67362c0b4ffc768bb4643dfb5efd0a5\": container with ID starting with 788ea62d7ba6f99ef9e2900c01c9a1bdb67362c0b4ffc768bb4643dfb5efd0a5 not found: ID does not exist" containerID="788ea62d7ba6f99ef9e2900c01c9a1bdb67362c0b4ffc768bb4643dfb5efd0a5"
Nov 01 06:03:08.990399 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:08.990327    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"788ea62d7ba6f99ef9e2900c01c9a1bdb67362c0b4ffc768bb4643dfb5efd0a5"} err="failed to get container status \"788ea62d7ba6f99ef9e2900c01c9a1bdb67362c0b4ffc768bb4643dfb5efd0a5\": rpc error: code = NotFound desc = could not find container \"788ea62d7ba6f99ef9e2900c01c9a1bdb67362c0b4ffc768bb4643dfb5efd0a5\": container with ID starting with 788ea62d7ba6f99ef9e2900c01c9a1bdb67362c0b4ffc768bb4643dfb5efd0a5 not found: ID does not exist"
Nov 01 06:03:09.149936 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:09.145093    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-c6e99c-bbf76fcb8-bbrnt"]
Nov 01 06:03:09.193592 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:09.193562    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-c6e99c-bbf76fcb8-bbrnt"]
Nov 01 06:03:09.482020 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:09.481981    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:10.067695 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:10.067663    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9729d894-945b-4d8b-8700-46d3552c2c8e" path="/var/lib/kubelet/pods/9729d894-945b-4d8b-8700-46d3552c2c8e/volumes"
Nov 01 06:03:11.587820 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:11.587790    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:11.987684 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:11.987647    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-c6e99c-8b7bf895f-hj225" podUID="7a44bf0d-d385-4d71-9b9a-b0c8d86f4c48" containerName="payload-processing" containerID="cri-o://0338c714995b2a99b57d637d8d23baaa0168a76cdd0d9e59e9e3e7bac49819bb" gracePeriod=25
Nov 01 06:03:12.093437 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:12.093399    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c6e99c-fcb666fb7-kc9cw" podUID="8420da2d-22af-4a07-9054-220111bd91bf" containerName="payload-pre-processing" containerID="cri-o://605ced9811813726efc1f6ee1752e066e1965dd7daed5d9160fceeb7ecae5bec" gracePeriod=25
Nov 01 06:03:12.700981 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:12.700947    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:12.880741 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:12.880707    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:14.034724 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:14.034695    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:14.236091 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:14.236058    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:14.737953 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:14.737923    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:15.204896 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:15.204868    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:16.036555 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:16.036517    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-c6e99c-8b7bf895f-hj225" podUID="7a44bf0d-d385-4d71-9b9a-b0c8d86f4c48" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:03:16.036752 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:16.036567    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c6e99c-fcb666fb7-kc9cw" podUID="8420da2d-22af-4a07-9054-220111bd91bf" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:03:16.715591 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:16.715560    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:19.191731 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:19.191700    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:19.445136 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:19.445064    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:20.156627 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:20.156598    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-a48099/simulator-w4-a48099-kserve-7df86f7d-rdlrm"]
Nov 01 06:03:20.276554 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:20.276524    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:20.790644 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:20.790607    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:21.293964 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:21.293934    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:21.887476 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:21.887415    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:22.058416 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:22.058387    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:26.036603 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:26.036558    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-c6e99c-8b7bf895f-hj225" podUID="7a44bf0d-d385-4d71-9b9a-b0c8d86f4c48" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:03:26.037051 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:26.036602    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c6e99c-fcb666fb7-kc9cw" podUID="8420da2d-22af-4a07-9054-220111bd91bf" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:03:26.329331 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:26.329259    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:27.042942 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:27.042878    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:28.113486 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:28.113443    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-afc555-84db8c5b8c-kh4bq"]
Nov 01 06:03:28.225445 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:28.225402    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-afc555-86567dff5-4gjlw"]
Nov 01 06:03:28.514152 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:28.514121    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:28.690980 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:28.690930    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:28.864607 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:28.864539    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:30.068798 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:03:30.068766    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 06:03:32.115807 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:32.115774    5689 generic.go:356] "Generic (PLEG): container finished" podID="7a44bf0d-d385-4d71-9b9a-b0c8d86f4c48" containerID="0338c714995b2a99b57d637d8d23baaa0168a76cdd0d9e59e9e3e7bac49819bb" exitCode=0
Nov 01 06:03:32.116110 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:32.115838    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-c6e99c-8b7bf895f-hj225" event={"ID":"7a44bf0d-d385-4d71-9b9a-b0c8d86f4c48","Type":"ContainerDied","Data":"0338c714995b2a99b57d637d8d23baaa0168a76cdd0d9e59e9e3e7bac49819bb"}
Nov 01 06:03:32.120839 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:32.120813    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:32.274335 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:32.274309    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-c6e99c-8b7bf895f-hj225"
Nov 01 06:03:32.363898 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:32.363875    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c6e99c-fcb666fb7-kc9cw"
Nov 01 06:03:32.396179 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:32.396145    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a44bf0d-d385-4d71-9b9a-b0c8d86f4c48-kube-api-access-r4j27\" (UniqueName: \"kubernetes.io/projected/7a44bf0d-d385-4d71-9b9a-b0c8d86f4c48-kube-api-access-r4j27\") pod \"7a44bf0d-d385-4d71-9b9a-b0c8d86f4c48\" (UID: \"7a44bf0d-d385-4d71-9b9a-b0c8d86f4c48\") "
Nov 01 06:03:32.396318 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:32.396212    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a44bf0d-d385-4d71-9b9a-b0c8d86f4c48-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a44bf0d-d385-4d71-9b9a-b0c8d86f4c48-plugins-config-volume\") pod \"7a44bf0d-d385-4d71-9b9a-b0c8d86f4c48\" (UID: \"7a44bf0d-d385-4d71-9b9a-b0c8d86f4c48\") "
Nov 01 06:03:32.396642 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:32.396612    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a44bf0d-d385-4d71-9b9a-b0c8d86f4c48-plugins-config-volume" pod "7a44bf0d-d385-4d71-9b9a-b0c8d86f4c48" (UID: "7a44bf0d-d385-4d71-9b9a-b0c8d86f4c48"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:03:32.396738 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:32.396713    5689 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a44bf0d-d385-4d71-9b9a-b0c8d86f4c48-plugins-config-volume\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:03:32.398495 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:32.398438    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a44bf0d-d385-4d71-9b9a-b0c8d86f4c48-kube-api-access-r4j27" pod "7a44bf0d-d385-4d71-9b9a-b0c8d86f4c48" (UID: "7a44bf0d-d385-4d71-9b9a-b0c8d86f4c48"). InnerVolumeSpecName "kube-api-access-r4j27". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:03:32.497072 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:32.497042    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8420da2d-22af-4a07-9054-220111bd91bf-kube-api-access-vmf76\" (UniqueName: \"kubernetes.io/projected/8420da2d-22af-4a07-9054-220111bd91bf-kube-api-access-vmf76\") pod \"8420da2d-22af-4a07-9054-220111bd91bf\" (UID: \"8420da2d-22af-4a07-9054-220111bd91bf\") "
Nov 01 06:03:32.497239 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:32.497211    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8420da2d-22af-4a07-9054-220111bd91bf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8420da2d-22af-4a07-9054-220111bd91bf-plugins-config-volume\") pod \"8420da2d-22af-4a07-9054-220111bd91bf\" (UID: \"8420da2d-22af-4a07-9054-220111bd91bf\") "
Nov 01 06:03:32.497630 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:32.497605    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4j27\" (UniqueName: \"kubernetes.io/projected/7a44bf0d-d385-4d71-9b9a-b0c8d86f4c48-kube-api-access-r4j27\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:03:32.497905 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:32.497633    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8420da2d-22af-4a07-9054-220111bd91bf-plugins-config-volume" pod "8420da2d-22af-4a07-9054-220111bd91bf" (UID: "8420da2d-22af-4a07-9054-220111bd91bf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:03:32.499382 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:32.499323    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8420da2d-22af-4a07-9054-220111bd91bf-kube-api-access-vmf76" pod "8420da2d-22af-4a07-9054-220111bd91bf" (UID: "8420da2d-22af-4a07-9054-220111bd91bf"). InnerVolumeSpecName "kube-api-access-vmf76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:03:32.598643 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:32.598610    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmf76\" (UniqueName: \"kubernetes.io/projected/8420da2d-22af-4a07-9054-220111bd91bf-kube-api-access-vmf76\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:03:32.598643 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:32.598637    5689 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8420da2d-22af-4a07-9054-220111bd91bf-plugins-config-volume\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:03:33.114381 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:33.114346    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-afc555-84db8c5b8c-kh4bq" podUID="28a2cf8f-22d0-426e-af53-a5c0128e8d5d" containerName="payload-processing" containerID="cri-o://f12935a2f8cc1a4c6fdeb66440cf38361ffac80ff3c227b6535a52179f4e3d08" gracePeriod=25
Nov 01 06:03:33.122912 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:33.122891    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-c6e99c-8b7bf895f-hj225"
Nov 01 06:03:33.123283 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:33.122923    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-c6e99c-8b7bf895f-hj225" event={"ID":"7a44bf0d-d385-4d71-9b9a-b0c8d86f4c48","Type":"ContainerDied","Data":"26da8973aea4dea3f4ec919ddc09a1056f26038941f945836e33589b4e7f9590"}
Nov 01 06:03:33.123283 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:33.122990    5689 scope.go:122] "RemoveContainer" containerID="0338c714995b2a99b57d637d8d23baaa0168a76cdd0d9e59e9e3e7bac49819bb"
Nov 01 06:03:33.124427 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:33.124389    5689 generic.go:356] "Generic (PLEG): container finished" podID="8420da2d-22af-4a07-9054-220111bd91bf" containerID="605ced9811813726efc1f6ee1752e066e1965dd7daed5d9160fceeb7ecae5bec" exitCode=0
Nov 01 06:03:33.124427 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:33.124430    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c6e99c-fcb666fb7-kc9cw"
Nov 01 06:03:33.124628 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:33.124487    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c6e99c-fcb666fb7-kc9cw" event={"ID":"8420da2d-22af-4a07-9054-220111bd91bf","Type":"ContainerDied","Data":"605ced9811813726efc1f6ee1752e066e1965dd7daed5d9160fceeb7ecae5bec"}
Nov 01 06:03:33.124628 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:33.124536    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c6e99c-fcb666fb7-kc9cw" event={"ID":"8420da2d-22af-4a07-9054-220111bd91bf","Type":"ContainerDied","Data":"c243794209dd0d46a520e557fd3aa951e0f752745fc495795f098b813677bd69"}
Nov 01 06:03:33.132296 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:33.132274    5689 scope.go:122] "RemoveContainer" containerID="605ced9811813726efc1f6ee1752e066e1965dd7daed5d9160fceeb7ecae5bec"
Nov 01 06:03:33.141582 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:33.141562    5689 scope.go:122] "RemoveContainer" containerID="605ced9811813726efc1f6ee1752e066e1965dd7daed5d9160fceeb7ecae5bec"
Nov 01 06:03:33.141825 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:03:33.141807    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"605ced9811813726efc1f6ee1752e066e1965dd7daed5d9160fceeb7ecae5bec\": container with ID starting with 605ced9811813726efc1f6ee1752e066e1965dd7daed5d9160fceeb7ecae5bec not found: ID does not exist" containerID="605ced9811813726efc1f6ee1752e066e1965dd7daed5d9160fceeb7ecae5bec"
Nov 01 06:03:33.141864 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:33.141832    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"605ced9811813726efc1f6ee1752e066e1965dd7daed5d9160fceeb7ecae5bec"} err="failed to get container status \"605ced9811813726efc1f6ee1752e066e1965dd7daed5d9160fceeb7ecae5bec\": rpc error: code = NotFound desc = could not find container \"605ced9811813726efc1f6ee1752e066e1965dd7daed5d9160fceeb7ecae5bec\": container with ID starting with 605ced9811813726efc1f6ee1752e066e1965dd7daed5d9160fceeb7ecae5bec not found: ID does not exist"
Nov 01 06:03:33.226378 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:33.226346    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-afc555-86567dff5-4gjlw" podUID="1ae03a27-0c9c-4ca2-89c1-678ea84e7d9d" containerName="payload-pre-processing" containerID="cri-o://faed5c03926ad4f9cfc8a8b6ce646c589802973b1a203bc7269056fc8c113dbb" gracePeriod=25
Nov 01 06:03:33.285866 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:33.285807    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-c6e99c-fcb666fb7-kc9cw"]
Nov 01 06:03:33.289642 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:33.289616    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-c6e99c-fcb666fb7-kc9cw"]
Nov 01 06:03:33.398500 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:33.398474    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-c6e99c-8b7bf895f-hj225"]
Nov 01 06:03:33.402700 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:33.402676    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-c6e99c-8b7bf895f-hj225"]
Nov 01 06:03:33.568592 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:33.568521    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:34.074020 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:34.073986    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a44bf0d-d385-4d71-9b9a-b0c8d86f4c48" path="/var/lib/kubelet/pods/7a44bf0d-d385-4d71-9b9a-b0c8d86f4c48/volumes"
Nov 01 06:03:34.075067 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:34.075042    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8420da2d-22af-4a07-9054-220111bd91bf" path="/var/lib/kubelet/pods/8420da2d-22af-4a07-9054-220111bd91bf/volumes"
Nov 01 06:03:34.888287 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:34.888254    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:35.181631 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:35.181599    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-a48099/simulator-w4-a48099-kserve-7df86f7d-rdlrm" podUID="12c3ad65-a804-429f-ab42-9cc7b2433855" containerName="main" containerID="cri-o://b2b3617cbf02448c6134ba0af81457476cb4268e819cb1c40c033436439e7178" gracePeriod=45
Nov 01 06:03:35.442488 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:35.442404    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-a48099/simulator-w4-a48099-kserve-7df86f7d-rdlrm"
Nov 01 06:03:35.536128 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:35.536100    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12c3ad65-a804-429f-ab42-9cc7b2433855-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12c3ad65-a804-429f-ab42-9cc7b2433855-tmp-dir\") pod \"12c3ad65-a804-429f-ab42-9cc7b2433855\" (UID: \"12c3ad65-a804-429f-ab42-9cc7b2433855\") "
Nov 01 06:03:35.536284 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:35.536148    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12c3ad65-a804-429f-ab42-9cc7b2433855-home\" (UniqueName: \"kubernetes.io/empty-dir/12c3ad65-a804-429f-ab42-9cc7b2433855-home\") pod \"12c3ad65-a804-429f-ab42-9cc7b2433855\" (UID: \"12c3ad65-a804-429f-ab42-9cc7b2433855\") "
Nov 01 06:03:35.536284 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:35.536188    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12c3ad65-a804-429f-ab42-9cc7b2433855-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12c3ad65-a804-429f-ab42-9cc7b2433855-model-cache\") pod \"12c3ad65-a804-429f-ab42-9cc7b2433855\" (UID: \"12c3ad65-a804-429f-ab42-9cc7b2433855\") "
Nov 01 06:03:35.536422 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:35.536318    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12c3ad65-a804-429f-ab42-9cc7b2433855-dshm\" (UniqueName: \"kubernetes.io/empty-dir/12c3ad65-a804-429f-ab42-9cc7b2433855-dshm\") pod \"12c3ad65-a804-429f-ab42-9cc7b2433855\" (UID: \"12c3ad65-a804-429f-ab42-9cc7b2433855\") "
Nov 01 06:03:35.536422 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:35.536355    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12c3ad65-a804-429f-ab42-9cc7b2433855-kube-api-access-kstcx\" (UniqueName: \"kubernetes.io/projected/12c3ad65-a804-429f-ab42-9cc7b2433855-kube-api-access-kstcx\") pod \"12c3ad65-a804-429f-ab42-9cc7b2433855\" (UID: \"12c3ad65-a804-429f-ab42-9cc7b2433855\") "
Nov 01 06:03:35.536422 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:35.536389    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12c3ad65-a804-429f-ab42-9cc7b2433855-tls-certs\" (UniqueName: \"kubernetes.io/secret/12c3ad65-a804-429f-ab42-9cc7b2433855-tls-certs\") pod \"12c3ad65-a804-429f-ab42-9cc7b2433855\" (UID: \"12c3ad65-a804-429f-ab42-9cc7b2433855\") "
Nov 01 06:03:35.536600 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:35.536435    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12c3ad65-a804-429f-ab42-9cc7b2433855-tmp-dir" pod "12c3ad65-a804-429f-ab42-9cc7b2433855" (UID: "12c3ad65-a804-429f-ab42-9cc7b2433855"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:03:35.536600 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:35.536464    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12c3ad65-a804-429f-ab42-9cc7b2433855-home" pod "12c3ad65-a804-429f-ab42-9cc7b2433855" (UID: "12c3ad65-a804-429f-ab42-9cc7b2433855"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:03:35.536600 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:35.536539    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12c3ad65-a804-429f-ab42-9cc7b2433855-model-cache" pod "12c3ad65-a804-429f-ab42-9cc7b2433855" (UID: "12c3ad65-a804-429f-ab42-9cc7b2433855"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:03:35.536888 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:35.536865    5689 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12c3ad65-a804-429f-ab42-9cc7b2433855-home\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:03:35.537000 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:35.536894    5689 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12c3ad65-a804-429f-ab42-9cc7b2433855-model-cache\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:03:35.537000 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:35.536910    5689 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12c3ad65-a804-429f-ab42-9cc7b2433855-tmp-dir\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:03:35.538903 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:35.538866    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12c3ad65-a804-429f-ab42-9cc7b2433855-tls-certs" pod "12c3ad65-a804-429f-ab42-9cc7b2433855" (UID: "12c3ad65-a804-429f-ab42-9cc7b2433855"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:03:35.539021 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:35.538972    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12c3ad65-a804-429f-ab42-9cc7b2433855-dshm" pod "12c3ad65-a804-429f-ab42-9cc7b2433855" (UID: "12c3ad65-a804-429f-ab42-9cc7b2433855"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:03:35.539217 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:35.539188    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12c3ad65-a804-429f-ab42-9cc7b2433855-kube-api-access-kstcx" pod "12c3ad65-a804-429f-ab42-9cc7b2433855" (UID: "12c3ad65-a804-429f-ab42-9cc7b2433855"). InnerVolumeSpecName "kube-api-access-kstcx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:03:35.638362 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:35.638334    5689 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12c3ad65-a804-429f-ab42-9cc7b2433855-dshm\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:03:35.638362 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:35.638364    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kstcx\" (UniqueName: \"kubernetes.io/projected/12c3ad65-a804-429f-ab42-9cc7b2433855-kube-api-access-kstcx\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:03:35.638574 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:35.638375    5689 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12c3ad65-a804-429f-ab42-9cc7b2433855-tls-certs\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:03:36.142477 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:36.142423    5689 generic.go:356] "Generic (PLEG): container finished" podID="12c3ad65-a804-429f-ab42-9cc7b2433855" containerID="b2b3617cbf02448c6134ba0af81457476cb4268e819cb1c40c033436439e7178" exitCode=0
Nov 01 06:03:36.142873 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:36.142504    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-a48099/simulator-w4-a48099-kserve-7df86f7d-rdlrm"
Nov 01 06:03:36.142873 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:36.142504    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-a48099/simulator-w4-a48099-kserve-7df86f7d-rdlrm" event={"ID":"12c3ad65-a804-429f-ab42-9cc7b2433855","Type":"ContainerDied","Data":"b2b3617cbf02448c6134ba0af81457476cb4268e819cb1c40c033436439e7178"}
Nov 01 06:03:36.142873 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:36.142615    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-a48099/simulator-w4-a48099-kserve-7df86f7d-rdlrm" event={"ID":"12c3ad65-a804-429f-ab42-9cc7b2433855","Type":"ContainerDied","Data":"57d0cf5226d7ee6af1047d3644a314b19dacb7f6ee0556ec7bb35dfb36979f32"}
Nov 01 06:03:36.142873 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:36.142633    5689 scope.go:122] "RemoveContainer" containerID="b2b3617cbf02448c6134ba0af81457476cb4268e819cb1c40c033436439e7178"
Nov 01 06:03:36.152553 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:36.152537    5689 scope.go:122] "RemoveContainer" containerID="b2b3617cbf02448c6134ba0af81457476cb4268e819cb1c40c033436439e7178"
Nov 01 06:03:36.152824 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:03:36.152805    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2b3617cbf02448c6134ba0af81457476cb4268e819cb1c40c033436439e7178\": container with ID starting with b2b3617cbf02448c6134ba0af81457476cb4268e819cb1c40c033436439e7178 not found: ID does not exist" containerID="b2b3617cbf02448c6134ba0af81457476cb4268e819cb1c40c033436439e7178"
Nov 01 06:03:36.152870 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:36.152831    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2b3617cbf02448c6134ba0af81457476cb4268e819cb1c40c033436439e7178"} err="failed to get container status \"b2b3617cbf02448c6134ba0af81457476cb4268e819cb1c40c033436439e7178\": rpc error: code = NotFound desc = could not find container \"b2b3617cbf02448c6134ba0af81457476cb4268e819cb1c40c033436439e7178\": container with ID starting with b2b3617cbf02448c6134ba0af81457476cb4268e819cb1c40c033436439e7178 not found: ID does not exist"
Nov 01 06:03:36.310426 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:36.310396    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-a48099/simulator-w4-a48099-kserve-7df86f7d-rdlrm"]
Nov 01 06:03:36.359803 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:36.359719    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-a48099/simulator-w4-a48099-kserve-7df86f7d-rdlrm"]
Nov 01 06:03:37.204160 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:37.204128    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:37.948624 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:37.948593    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:38.067334 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:38.067306    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12c3ad65-a804-429f-ab42-9cc7b2433855" path="/var/lib/kubelet/pods/12c3ad65-a804-429f-ab42-9cc7b2433855/volumes"
Nov 01 06:03:39.465805 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:39.465771    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-afc555-84db8c5b8c-kh4bq" podUID="28a2cf8f-22d0-426e-af53-a5c0128e8d5d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:03:39.466170 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:39.465772    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-afc555-86567dff5-4gjlw" podUID="1ae03a27-0c9c-4ca2-89c1-678ea84e7d9d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:03:41.331220 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:41.331181    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:43.119699 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:43.119606    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:43.347229 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:43.347203    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:03:43.831380 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:43.831352    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:45.391956 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:45.391921    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:45.574079 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:45.574048    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:46.999715 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:46.999672    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:47.536785 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:47.536753    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:48.162379 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:48.162348    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:49.465509 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:49.465477    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-afc555-86567dff5-4gjlw" podUID="1ae03a27-0c9c-4ca2-89c1-678ea84e7d9d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:03:49.465934 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:49.465480    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-afc555-84db8c5b8c-kh4bq" podUID="28a2cf8f-22d0-426e-af53-a5c0128e8d5d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:03:52.415566 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:52.415471    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-afc555-d4c6bcc8f-fnlj6"]
Nov 01 06:03:52.416012 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:52.415737    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-afc555-d4c6bcc8f-fnlj6" podUID="0226e80d-bc64-44a1-86cf-cde37ac9277c" containerName="maas-api" containerID="cri-o://f3679e4fd5ce7d1783402079b40f384abd5d3b565d34ca21e6feb1de5b26927b" gracePeriod=30
Nov 01 06:03:52.553397 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:52.553364    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:52.702139 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:52.702118    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-afc555-d4c6bcc8f-fnlj6"
Nov 01 06:03:52.733185 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:52.733159    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0226e80d-bc64-44a1-86cf-cde37ac9277c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0226e80d-bc64-44a1-86cf-cde37ac9277c-maas-api-tls\") pod \"0226e80d-bc64-44a1-86cf-cde37ac9277c\" (UID: \"0226e80d-bc64-44a1-86cf-cde37ac9277c\") "
Nov 01 06:03:52.733305 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:52.733238    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0226e80d-bc64-44a1-86cf-cde37ac9277c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0226e80d-bc64-44a1-86cf-cde37ac9277c-openshift-service-ca\") pod \"0226e80d-bc64-44a1-86cf-cde37ac9277c\" (UID: \"0226e80d-bc64-44a1-86cf-cde37ac9277c\") "
Nov 01 06:03:52.733305 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:52.733272    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0226e80d-bc64-44a1-86cf-cde37ac9277c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0226e80d-bc64-44a1-86cf-cde37ac9277c-metrics-certs\") pod \"0226e80d-bc64-44a1-86cf-cde37ac9277c\" (UID: \"0226e80d-bc64-44a1-86cf-cde37ac9277c\") "
Nov 01 06:03:52.733417 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:52.733307    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0226e80d-bc64-44a1-86cf-cde37ac9277c-kube-api-access-btx6p\" (UniqueName: \"kubernetes.io/projected/0226e80d-bc64-44a1-86cf-cde37ac9277c-kube-api-access-btx6p\") pod \"0226e80d-bc64-44a1-86cf-cde37ac9277c\" (UID: \"0226e80d-bc64-44a1-86cf-cde37ac9277c\") "
Nov 01 06:03:52.733630 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:52.733602    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0226e80d-bc64-44a1-86cf-cde37ac9277c-openshift-service-ca" pod "0226e80d-bc64-44a1-86cf-cde37ac9277c" (UID: "0226e80d-bc64-44a1-86cf-cde37ac9277c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:03:52.735316 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:52.735294    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0226e80d-bc64-44a1-86cf-cde37ac9277c-maas-api-tls" pod "0226e80d-bc64-44a1-86cf-cde37ac9277c" (UID: "0226e80d-bc64-44a1-86cf-cde37ac9277c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:03:52.735510 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:52.735488    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0226e80d-bc64-44a1-86cf-cde37ac9277c-kube-api-access-btx6p" pod "0226e80d-bc64-44a1-86cf-cde37ac9277c" (UID: "0226e80d-bc64-44a1-86cf-cde37ac9277c"). InnerVolumeSpecName "kube-api-access-btx6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:03:52.735577 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:52.735527    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0226e80d-bc64-44a1-86cf-cde37ac9277c-metrics-certs" pod "0226e80d-bc64-44a1-86cf-cde37ac9277c" (UID: "0226e80d-bc64-44a1-86cf-cde37ac9277c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:03:52.834292 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:52.834259    5689 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0226e80d-bc64-44a1-86cf-cde37ac9277c-openshift-service-ca\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:03:52.834292 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:52.834290    5689 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0226e80d-bc64-44a1-86cf-cde37ac9277c-metrics-certs\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:03:52.834513 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:52.834303    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btx6p\" (UniqueName: \"kubernetes.io/projected/0226e80d-bc64-44a1-86cf-cde37ac9277c-kube-api-access-btx6p\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:03:52.834513 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:52.834316    5689 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0226e80d-bc64-44a1-86cf-cde37ac9277c-maas-api-tls\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:03:53.241410 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:53.241384    5689 generic.go:356] "Generic (PLEG): container finished" podID="0226e80d-bc64-44a1-86cf-cde37ac9277c" containerID="f3679e4fd5ce7d1783402079b40f384abd5d3b565d34ca21e6feb1de5b26927b" exitCode=0
Nov 01 06:03:53.241563 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:53.241424    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-afc555-d4c6bcc8f-fnlj6"
Nov 01 06:03:53.242073 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:53.242043    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-afc555-d4c6bcc8f-fnlj6" event={"ID":"0226e80d-bc64-44a1-86cf-cde37ac9277c","Type":"ContainerDied","Data":"f3679e4fd5ce7d1783402079b40f384abd5d3b565d34ca21e6feb1de5b26927b"}
Nov 01 06:03:53.242182 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:53.242113    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-afc555-d4c6bcc8f-fnlj6" event={"ID":"0226e80d-bc64-44a1-86cf-cde37ac9277c","Type":"ContainerDied","Data":"c782a9dbda6dbb1d16b252b693b8136497d159689d537ae847586088de98fc3e"}
Nov 01 06:03:53.242182 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:53.242149    5689 scope.go:122] "RemoveContainer" containerID="f3679e4fd5ce7d1783402079b40f384abd5d3b565d34ca21e6feb1de5b26927b"
Nov 01 06:03:53.243491 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:53.243448    5689 generic.go:356] "Generic (PLEG): container finished" podID="28a2cf8f-22d0-426e-af53-a5c0128e8d5d" containerID="f12935a2f8cc1a4c6fdeb66440cf38361ffac80ff3c227b6535a52179f4e3d08" exitCode=0
Nov 01 06:03:53.243491 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:53.243487    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-afc555-84db8c5b8c-kh4bq" event={"ID":"28a2cf8f-22d0-426e-af53-a5c0128e8d5d","Type":"ContainerDied","Data":"f12935a2f8cc1a4c6fdeb66440cf38361ffac80ff3c227b6535a52179f4e3d08"}
Nov 01 06:03:53.340747 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:53.340719    5689 scope.go:122] "RemoveContainer" containerID="f3679e4fd5ce7d1783402079b40f384abd5d3b565d34ca21e6feb1de5b26927b"
Nov 01 06:03:53.341202 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:03:53.341168    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3679e4fd5ce7d1783402079b40f384abd5d3b565d34ca21e6feb1de5b26927b\": container with ID starting with f3679e4fd5ce7d1783402079b40f384abd5d3b565d34ca21e6feb1de5b26927b not found: ID does not exist" containerID="f3679e4fd5ce7d1783402079b40f384abd5d3b565d34ca21e6feb1de5b26927b"
Nov 01 06:03:53.341317 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:53.341214    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3679e4fd5ce7d1783402079b40f384abd5d3b565d34ca21e6feb1de5b26927b"} err="failed to get container status \"f3679e4fd5ce7d1783402079b40f384abd5d3b565d34ca21e6feb1de5b26927b\": rpc error: code = NotFound desc = could not find container \"f3679e4fd5ce7d1783402079b40f384abd5d3b565d34ca21e6feb1de5b26927b\": container with ID starting with f3679e4fd5ce7d1783402079b40f384abd5d3b565d34ca21e6feb1de5b26927b not found: ID does not exist"
Nov 01 06:03:53.366311 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:53.366291    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-afc555-84db8c5b8c-kh4bq"
Nov 01 06:03:53.463915 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:53.458108    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-afc555-d4c6bcc8f-fnlj6"]
Nov 01 06:03:53.466023 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:53.465996    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-afc555-d4c6bcc8f-fnlj6"]
Nov 01 06:03:53.548932 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:53.548909    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-afc555-86567dff5-4gjlw"
Nov 01 06:03:53.558356 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:53.558329    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28a2cf8f-22d0-426e-af53-a5c0128e8d5d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28a2cf8f-22d0-426e-af53-a5c0128e8d5d-plugins-config-volume\") pod \"28a2cf8f-22d0-426e-af53-a5c0128e8d5d\" (UID: \"28a2cf8f-22d0-426e-af53-a5c0128e8d5d\") "
Nov 01 06:03:53.558487 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:53.558385    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28a2cf8f-22d0-426e-af53-a5c0128e8d5d-kube-api-access-m2zk8\" (UniqueName: \"kubernetes.io/projected/28a2cf8f-22d0-426e-af53-a5c0128e8d5d-kube-api-access-m2zk8\") pod \"28a2cf8f-22d0-426e-af53-a5c0128e8d5d\" (UID: \"28a2cf8f-22d0-426e-af53-a5c0128e8d5d\") "
Nov 01 06:03:53.558553 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:53.558536    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ae03a27-0c9c-4ca2-89c1-678ea84e7d9d-kube-api-access-5h8pq\" (UniqueName: \"kubernetes.io/projected/1ae03a27-0c9c-4ca2-89c1-678ea84e7d9d-kube-api-access-5h8pq\") pod \"1ae03a27-0c9c-4ca2-89c1-678ea84e7d9d\" (UID: \"1ae03a27-0c9c-4ca2-89c1-678ea84e7d9d\") "
Nov 01 06:03:53.558805 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:53.558778    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28a2cf8f-22d0-426e-af53-a5c0128e8d5d-plugins-config-volume" pod "28a2cf8f-22d0-426e-af53-a5c0128e8d5d" (UID: "28a2cf8f-22d0-426e-af53-a5c0128e8d5d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:03:53.558965 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:53.558946    5689 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28a2cf8f-22d0-426e-af53-a5c0128e8d5d-plugins-config-volume\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:03:53.560527 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:53.560501    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28a2cf8f-22d0-426e-af53-a5c0128e8d5d-kube-api-access-m2zk8" pod "28a2cf8f-22d0-426e-af53-a5c0128e8d5d" (UID: "28a2cf8f-22d0-426e-af53-a5c0128e8d5d"). InnerVolumeSpecName "kube-api-access-m2zk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:03:53.560689 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:53.560671    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ae03a27-0c9c-4ca2-89c1-678ea84e7d9d-kube-api-access-5h8pq" pod "1ae03a27-0c9c-4ca2-89c1-678ea84e7d9d" (UID: "1ae03a27-0c9c-4ca2-89c1-678ea84e7d9d"). InnerVolumeSpecName "kube-api-access-5h8pq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:03:53.659390 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:53.659361    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ae03a27-0c9c-4ca2-89c1-678ea84e7d9d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ae03a27-0c9c-4ca2-89c1-678ea84e7d9d-plugins-config-volume\") pod \"1ae03a27-0c9c-4ca2-89c1-678ea84e7d9d\" (UID: \"1ae03a27-0c9c-4ca2-89c1-678ea84e7d9d\") "
Nov 01 06:03:53.659666 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:53.659652    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5h8pq\" (UniqueName: \"kubernetes.io/projected/1ae03a27-0c9c-4ca2-89c1-678ea84e7d9d-kube-api-access-5h8pq\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:03:53.659717 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:53.659671    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2zk8\" (UniqueName: \"kubernetes.io/projected/28a2cf8f-22d0-426e-af53-a5c0128e8d5d-kube-api-access-m2zk8\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:03:53.659717 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:53.659696    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ae03a27-0c9c-4ca2-89c1-678ea84e7d9d-plugins-config-volume" pod "1ae03a27-0c9c-4ca2-89c1-678ea84e7d9d" (UID: "1ae03a27-0c9c-4ca2-89c1-678ea84e7d9d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:03:53.760649 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:53.760575    5689 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ae03a27-0c9c-4ca2-89c1-678ea84e7d9d-plugins-config-volume\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:03:54.067633 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:54.067562    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0226e80d-bc64-44a1-86cf-cde37ac9277c" path="/var/lib/kubelet/pods/0226e80d-bc64-44a1-86cf-cde37ac9277c/volumes"
Nov 01 06:03:54.252413 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:54.252372    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-afc555-84db8c5b8c-kh4bq"
Nov 01 06:03:54.252413 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:54.252399    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-afc555-84db8c5b8c-kh4bq" event={"ID":"28a2cf8f-22d0-426e-af53-a5c0128e8d5d","Type":"ContainerDied","Data":"d38323f33d9a0b26ff3bf263347003f4aced9e82a01a5658c46d1b32c471709e"}
Nov 01 06:03:54.252671 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:54.252478    5689 scope.go:122] "RemoveContainer" containerID="f12935a2f8cc1a4c6fdeb66440cf38361ffac80ff3c227b6535a52179f4e3d08"
Nov 01 06:03:54.254106 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:54.254078    5689 generic.go:356] "Generic (PLEG): container finished" podID="1ae03a27-0c9c-4ca2-89c1-678ea84e7d9d" containerID="faed5c03926ad4f9cfc8a8b6ce646c589802973b1a203bc7269056fc8c113dbb" exitCode=0
Nov 01 06:03:54.254206 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:54.254121    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-afc555-86567dff5-4gjlw"
Nov 01 06:03:54.254206 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:54.254133    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-afc555-86567dff5-4gjlw" event={"ID":"1ae03a27-0c9c-4ca2-89c1-678ea84e7d9d","Type":"ContainerDied","Data":"faed5c03926ad4f9cfc8a8b6ce646c589802973b1a203bc7269056fc8c113dbb"}
Nov 01 06:03:54.254206 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:54.254174    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-afc555-86567dff5-4gjlw" event={"ID":"1ae03a27-0c9c-4ca2-89c1-678ea84e7d9d","Type":"ContainerDied","Data":"893900e7bcc1234775edefebe9164720b0bc810c8966cc15146955429c41f5d9"}
Nov 01 06:03:54.262483 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:54.262440    5689 scope.go:122] "RemoveContainer" containerID="faed5c03926ad4f9cfc8a8b6ce646c589802973b1a203bc7269056fc8c113dbb"
Nov 01 06:03:54.270635 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:54.270617    5689 scope.go:122] "RemoveContainer" containerID="faed5c03926ad4f9cfc8a8b6ce646c589802973b1a203bc7269056fc8c113dbb"
Nov 01 06:03:54.270915 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:03:54.270897    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"faed5c03926ad4f9cfc8a8b6ce646c589802973b1a203bc7269056fc8c113dbb\": container with ID starting with faed5c03926ad4f9cfc8a8b6ce646c589802973b1a203bc7269056fc8c113dbb not found: ID does not exist" containerID="faed5c03926ad4f9cfc8a8b6ce646c589802973b1a203bc7269056fc8c113dbb"
Nov 01 06:03:54.270969 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:54.270923    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faed5c03926ad4f9cfc8a8b6ce646c589802973b1a203bc7269056fc8c113dbb"} err="failed to get container status \"faed5c03926ad4f9cfc8a8b6ce646c589802973b1a203bc7269056fc8c113dbb\": rpc error: code = NotFound desc = could not find container \"faed5c03926ad4f9cfc8a8b6ce646c589802973b1a203bc7269056fc8c113dbb\": container with ID starting with faed5c03926ad4f9cfc8a8b6ce646c589802973b1a203bc7269056fc8c113dbb not found: ID does not exist"
Nov 01 06:03:54.414410 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:54.414381    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-afc555-84db8c5b8c-kh4bq"]
Nov 01 06:03:54.418778 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:54.418755    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-afc555-84db8c5b8c-kh4bq"]
Nov 01 06:03:54.446576 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:54.446553    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:54.528567 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:54.528530    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-afc555-86567dff5-4gjlw"]
Nov 01 06:03:54.533082 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:54.533054    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-afc555-86567dff5-4gjlw"]
Nov 01 06:03:56.067564 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:56.067530    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ae03a27-0c9c-4ca2-89c1-678ea84e7d9d" path="/var/lib/kubelet/pods/1ae03a27-0c9c-4ca2-89c1-678ea84e7d9d/volumes"
Nov 01 06:03:56.067932 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:56.067898    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28a2cf8f-22d0-426e-af53-a5c0128e8d5d" path="/var/lib/kubelet/pods/28a2cf8f-22d0-426e-af53-a5c0128e8d5d/volumes"
Nov 01 06:03:57.226749 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:57.226714    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:57.456215 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:57.456184    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:57.636127 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:57.636055    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:58.564059 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:58.564027    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:59.134695 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:59.134661    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:59.314936 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:59.314901    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:59.500064 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:03:59.500033    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:04:00.068781 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:04:00.068752    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 06:04:00.613395 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:00.613362    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:04:01.139936 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:01.139865    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:04:01.917064 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:01.917032    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:04:03.725258 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:03.725229    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:04:04.370427 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:04.370385    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:04:04.582501 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:04.577331    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:04:05.591690 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:05.591659    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:04:08.222964 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:08.222933    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:04:08.546851 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:08.546779    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:04:08.718339 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:08.718301    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:04:08.832673 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:08.832597    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gk2pd"]
Nov 01 06:04:08.833128 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:08.833111    5689 cpu_manager.go:414] "containerMap: removing container" podUID="7a44bf0d-d385-4d71-9b9a-b0c8d86f4c48" containerName="payload-processing"
Nov 01 06:04:08.833171 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:08.833131    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a44bf0d-d385-4d71-9b9a-b0c8d86f4c48" containerName="payload-processing"
Nov 01 06:04:08.833171 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:08.833143    5689 cpu_manager.go:414] "containerMap: removing container" podUID="ef1eafdc-0b49-42ed-ac95-6bddfd5486ec" containerName="maas-api"
Nov 01 06:04:08.833171 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:08.833148    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef1eafdc-0b49-42ed-ac95-6bddfd5486ec" containerName="maas-api"
Nov 01 06:04:08.833171 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:08.833157    5689 cpu_manager.go:414] "containerMap: removing container" podUID="0226e80d-bc64-44a1-86cf-cde37ac9277c" containerName="maas-api"
Nov 01 06:04:08.833171 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:08.833163    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0226e80d-bc64-44a1-86cf-cde37ac9277c" containerName="maas-api"
Nov 01 06:04:08.833330 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:08.833175    5689 cpu_manager.go:414] "containerMap: removing container" podUID="12c3ad65-a804-429f-ab42-9cc7b2433855" containerName="main"
Nov 01 06:04:08.833330 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:08.833181    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12c3ad65-a804-429f-ab42-9cc7b2433855" containerName="main"
Nov 01 06:04:08.833330 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:08.833192    5689 cpu_manager.go:414] "containerMap: removing container" podUID="28a2cf8f-22d0-426e-af53-a5c0128e8d5d" containerName="payload-processing"
Nov 01 06:04:08.833330 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:08.833199    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28a2cf8f-22d0-426e-af53-a5c0128e8d5d" containerName="payload-processing"
Nov 01 06:04:08.833330 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:08.833208    5689 cpu_manager.go:414] "containerMap: removing container" podUID="838940c1-785d-4fb4-9c0a-c967b252ff64" containerName="payload-processing"
Nov 01 06:04:08.833330 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:08.833212    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="838940c1-785d-4fb4-9c0a-c967b252ff64" containerName="payload-processing"
Nov 01 06:04:08.833330 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:08.833220    5689 cpu_manager.go:414] "containerMap: removing container" podUID="9729d894-945b-4d8b-8700-46d3552c2c8e" containerName="maas-api"
Nov 01 06:04:08.833330 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:08.833224    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9729d894-945b-4d8b-8700-46d3552c2c8e" containerName="maas-api"
Nov 01 06:04:08.833330 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:08.833242    5689 cpu_manager.go:414] "containerMap: removing container" podUID="1177a649-2110-4dae-86a9-dc801392d954" containerName="payload-pre-processing"
Nov 01 06:04:08.833330 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:08.833246    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1177a649-2110-4dae-86a9-dc801392d954" containerName="payload-pre-processing"
Nov 01 06:04:08.833330 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:08.833255    5689 cpu_manager.go:414] "containerMap: removing container" podUID="8420da2d-22af-4a07-9054-220111bd91bf" containerName="payload-pre-processing"
Nov 01 06:04:08.833330 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:08.833260    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8420da2d-22af-4a07-9054-220111bd91bf" containerName="payload-pre-processing"
Nov 01 06:04:08.833330 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:08.833273    5689 cpu_manager.go:414] "containerMap: removing container" podUID="1ae03a27-0c9c-4ca2-89c1-678ea84e7d9d" containerName="payload-pre-processing"
Nov 01 06:04:08.833330 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:08.833280    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ae03a27-0c9c-4ca2-89c1-678ea84e7d9d" containerName="payload-pre-processing"
Nov 01 06:04:08.833809 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:08.833348    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="9729d894-945b-4d8b-8700-46d3552c2c8e" containerName="maas-api"
Nov 01 06:04:08.833809 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:08.833364    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="1177a649-2110-4dae-86a9-dc801392d954" containerName="payload-pre-processing"
Nov 01 06:04:08.833809 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:08.833369    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="8420da2d-22af-4a07-9054-220111bd91bf" containerName="payload-pre-processing"
Nov 01 06:04:08.833809 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:08.833377    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ae03a27-0c9c-4ca2-89c1-678ea84e7d9d" containerName="payload-pre-processing"
Nov 01 06:04:08.833809 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:08.833385    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a44bf0d-d385-4d71-9b9a-b0c8d86f4c48" containerName="payload-processing"
Nov 01 06:04:08.833809 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:08.833390    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef1eafdc-0b49-42ed-ac95-6bddfd5486ec" containerName="maas-api"
Nov 01 06:04:08.833809 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:08.833395    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="0226e80d-bc64-44a1-86cf-cde37ac9277c" containerName="maas-api"
Nov 01 06:04:08.833809 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:08.833402    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="12c3ad65-a804-429f-ab42-9cc7b2433855" containerName="main"
Nov 01 06:04:08.833809 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:08.833406    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="28a2cf8f-22d0-426e-af53-a5c0128e8d5d" containerName="payload-processing"
Nov 01 06:04:08.833809 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:08.833412    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="838940c1-785d-4fb4-9c0a-c967b252ff64" containerName="payload-processing"
Nov 01 06:04:08.836717 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:08.836699    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gk2pd"
Nov 01 06:04:08.922182 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:08.922150    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gk2pd"]
Nov 01 06:04:08.938645 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:08.938616    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2-istio-data\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-gk2pd\" (UID: \"3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gk2pd"
Nov 01 06:04:08.938894 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:08.938863    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2-workload-certs\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-gk2pd\" (UID: \"3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gk2pd"
Nov 01 06:04:08.939047 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:08.938911    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2-istio-token\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-gk2pd\" (UID: \"3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gk2pd"
Nov 01 06:04:08.939047 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:08.938979    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6429\" (UniqueName: \"kubernetes.io/projected/3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2-kube-api-access-p6429\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-gk2pd\" (UID: \"3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gk2pd"
Nov 01 06:04:08.939178 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:08.939115    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2-istio-envoy\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-gk2pd\" (UID: \"3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gk2pd"
Nov 01 06:04:08.939311 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:08.939270    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2-istio-podinfo\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-gk2pd\" (UID: \"3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gk2pd"
Nov 01 06:04:08.939381 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:08.939333    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2-workload-socket\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-gk2pd\" (UID: \"3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gk2pd"
Nov 01 06:04:08.939381 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:08.939374    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2-credential-socket\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-gk2pd\" (UID: \"3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gk2pd"
Nov 01 06:04:08.939530 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:08.939399    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2-istiod-ca-cert\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-gk2pd\" (UID: \"3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gk2pd"
Nov 01 06:04:09.039952 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:09.039915    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2-istio-envoy\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-gk2pd\" (UID: \"3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gk2pd"
Nov 01 06:04:09.040156 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:09.039980    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2-istio-podinfo\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-gk2pd\" (UID: \"3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gk2pd"
Nov 01 06:04:09.040156 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:09.040011    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2-workload-socket\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-gk2pd\" (UID: \"3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gk2pd"
Nov 01 06:04:09.040156 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:09.040034    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2-credential-socket\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-gk2pd\" (UID: \"3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gk2pd"
Nov 01 06:04:09.040156 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:09.040059    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2-istiod-ca-cert\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-gk2pd\" (UID: \"3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gk2pd"
Nov 01 06:04:09.040156 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:09.040094    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2-istio-data\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-gk2pd\" (UID: \"3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gk2pd"
Nov 01 06:04:09.040156 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:09.040132    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2-workload-certs\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-gk2pd\" (UID: \"3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gk2pd"
Nov 01 06:04:09.040156 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:09.040154    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2-istio-token\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-gk2pd\" (UID: \"3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gk2pd"
Nov 01 06:04:09.040583 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:09.040209    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6429\" (UniqueName: \"kubernetes.io/projected/3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2-kube-api-access-p6429\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-gk2pd\" (UID: \"3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gk2pd"
Nov 01 06:04:09.040583 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:09.040563    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2-credential-socket\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-gk2pd\" (UID: \"3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gk2pd"
Nov 01 06:04:09.040754 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:09.040709    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2-workload-certs\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-gk2pd\" (UID: \"3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gk2pd"
Nov 01 06:04:09.040754 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:09.040735    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2-istio-data\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-gk2pd\" (UID: \"3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gk2pd"
Nov 01 06:04:09.041015 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:09.040846    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2-istiod-ca-cert\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-gk2pd\" (UID: \"3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gk2pd"
Nov 01 06:04:09.041015 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:09.040991    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2-workload-socket\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-gk2pd\" (UID: \"3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gk2pd"
Nov 01 06:04:09.042577 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:09.042556    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2-istio-envoy\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-gk2pd\" (UID: \"3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gk2pd"
Nov 01 06:04:09.042699 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:09.042651    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2-istio-podinfo\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-gk2pd\" (UID: \"3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gk2pd"
Nov 01 06:04:09.101033 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:09.100969    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2-istio-token\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-gk2pd\" (UID: \"3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gk2pd"
Nov 01 06:04:09.101313 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:09.101295    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6429\" (UniqueName: \"kubernetes.io/projected/3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2-kube-api-access-p6429\") pod \"e2e-worker-w4-a48099-openshift-default-577864759-gk2pd\" (UID: \"3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2\") " pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gk2pd"
Nov 01 06:04:09.149215 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:09.149171    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gk2pd"
Nov 01 06:04:09.411683 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:09.411650    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:04:09.579714 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:09.579690    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gk2pd"]
Nov 01 06:04:09.582576 ip-10-0-129-92 kubenswrapper[5689]: W1101 06:04:09.582545    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fb62ef2_98a5_42ec_95ce_f18c1f3f81b2.slice/crio-7545c6f44257cd1a7b6703e9605f55e620ce93f2bf69c32e97557011fb81264a WatchSource:0}: Error finding container 7545c6f44257cd1a7b6703e9605f55e620ce93f2bf69c32e97557011fb81264a: Status 404 returned error can't find the container with id 7545c6f44257cd1a7b6703e9605f55e620ce93f2bf69c32e97557011fb81264a
Nov 01 06:04:09.584918 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:09.584889    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 06:04:09.584990 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:09.584953    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 06:04:09.584990 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:09.584982    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 06:04:09.595061 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:09.595034    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:04:10.036182 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:10.036125    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:04:10.347956 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:10.347882    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gk2pd" event={"ID":"3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2","Type":"ContainerStarted","Data":"7f8e1ce9f56c382921c26ab4727315b9bdcd2dafbc1fa1337085dbb0a20c4ab2"}
Nov 01 06:04:10.347956 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:10.347934    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gk2pd" event={"ID":"3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2","Type":"ContainerStarted","Data":"7545c6f44257cd1a7b6703e9605f55e620ce93f2bf69c32e97557011fb81264a"}
Nov 01 06:04:10.512036 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:10.511949    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gk2pd"]
Nov 01 06:04:10.523741 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:10.523706    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-5bms2"]
Nov 01 06:04:10.524314 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:10.524258    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-5bms2" podUID="ea6d76c0-414c-4ae6-9c9a-01e41971592f" containerName="istio-proxy" containerID="cri-o://c8321bede367ae90e95bf2919048be4282a992a0198f64bca75abbd5e02669cf" gracePeriod=30
Nov 01 06:04:10.526015 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:10.525997    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-fchpl"]
Nov 01 06:04:10.526175 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:10.526159    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-fchpl" podUID="9001d49f-9167-48e7-bacd-91219748d5e1" containerName="istio-proxy" containerID="cri-o://0062d22ba7f3f9526e542d6c2277e43627d76594563c663faa3e88bf44b73191" gracePeriod=30
Nov 01 06:04:10.540032 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:10.540009    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:04:10.543626 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:10.543588    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gk2pd" podStartSLOduration=2.543576362 podStartE2EDuration="2.543576362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:04:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:04:10.509851103 +0000 UTC m=+3628.580647825" watchObservedRunningTime="2026-11-01 06:04:10.543576362 +0000 UTC m=+3628.614373082"
Nov 01 06:04:10.761805 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:10.758380    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:04:11.149576 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:11.149551    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gk2pd"
Nov 01 06:04:11.150712 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:11.150686    5689 patch_prober.go:29] interesting pod/e2e-worker-w4-a48099-openshift-default-577864759-gk2pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Nov 01 06:04:11.150848 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:11.150734    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gk2pd" podUID="3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Nov 01 06:04:11.361438 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:11.361402    5689 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gk2pd" secret="" err="secret \"e2e-worker-w4-a48099-openshift-default-dockercfg-8mhxp\" not found"
Nov 01 06:04:12.150288 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:12.150258    5689 patch_prober.go:29] interesting pod/e2e-worker-w4-a48099-openshift-default-577864759-gk2pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Nov 01 06:04:12.150694 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:12.150305    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gk2pd" podUID="3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Nov 01 06:04:12.366437 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:12.366396    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gk2pd" podUID="3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2" containerName="istio-proxy" containerID="cri-o://7f8e1ce9f56c382921c26ab4727315b9bdcd2dafbc1fa1337085dbb0a20c4ab2" gracePeriod=30
Nov 01 06:04:13.904267 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:13.904209    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:04:14.400431 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:14.400400    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:04:15.795120 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:15.795100    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-5bms2"
Nov 01 06:04:15.810014 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:15.809993    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-fchpl"
Nov 01 06:04:15.950734 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:15.950706    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9001d49f-9167-48e7-bacd-91219748d5e1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9001d49f-9167-48e7-bacd-91219748d5e1-istio-envoy\") pod \"9001d49f-9167-48e7-bacd-91219748d5e1\" (UID: \"9001d49f-9167-48e7-bacd-91219748d5e1\") "
Nov 01 06:04:15.950881 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:15.950749    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea6d76c0-414c-4ae6-9c9a-01e41971592f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea6d76c0-414c-4ae6-9c9a-01e41971592f-istiod-ca-cert\") pod \"ea6d76c0-414c-4ae6-9c9a-01e41971592f\" (UID: \"ea6d76c0-414c-4ae6-9c9a-01e41971592f\") "
Nov 01 06:04:15.950881 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:15.950772    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9001d49f-9167-48e7-bacd-91219748d5e1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9001d49f-9167-48e7-bacd-91219748d5e1-workload-certs\") pod \"9001d49f-9167-48e7-bacd-91219748d5e1\" (UID: \"9001d49f-9167-48e7-bacd-91219748d5e1\") "
Nov 01 06:04:15.950881 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:15.950808    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea6d76c0-414c-4ae6-9c9a-01e41971592f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea6d76c0-414c-4ae6-9c9a-01e41971592f-credential-socket\") pod \"ea6d76c0-414c-4ae6-9c9a-01e41971592f\" (UID: \"ea6d76c0-414c-4ae6-9c9a-01e41971592f\") "
Nov 01 06:04:15.950881 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:15.950850    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9001d49f-9167-48e7-bacd-91219748d5e1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9001d49f-9167-48e7-bacd-91219748d5e1-workload-socket\") pod \"9001d49f-9167-48e7-bacd-91219748d5e1\" (UID: \"9001d49f-9167-48e7-bacd-91219748d5e1\") "
Nov 01 06:04:15.951109 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:15.950886    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9001d49f-9167-48e7-bacd-91219748d5e1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9001d49f-9167-48e7-bacd-91219748d5e1-credential-socket\") pod \"9001d49f-9167-48e7-bacd-91219748d5e1\" (UID: \"9001d49f-9167-48e7-bacd-91219748d5e1\") "
Nov 01 06:04:15.951109 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:15.950923    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea6d76c0-414c-4ae6-9c9a-01e41971592f-istio-token\" (UniqueName: \"kubernetes.io/projected/ea6d76c0-414c-4ae6-9c9a-01e41971592f-istio-token\") pod \"ea6d76c0-414c-4ae6-9c9a-01e41971592f\" (UID: \"ea6d76c0-414c-4ae6-9c9a-01e41971592f\") "
Nov 01 06:04:15.951109 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:15.950959    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea6d76c0-414c-4ae6-9c9a-01e41971592f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea6d76c0-414c-4ae6-9c9a-01e41971592f-istio-envoy\") pod \"ea6d76c0-414c-4ae6-9c9a-01e41971592f\" (UID: \"ea6d76c0-414c-4ae6-9c9a-01e41971592f\") "
Nov 01 06:04:15.951109 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:15.950984    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9001d49f-9167-48e7-bacd-91219748d5e1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9001d49f-9167-48e7-bacd-91219748d5e1-istio-podinfo\") pod \"9001d49f-9167-48e7-bacd-91219748d5e1\" (UID: \"9001d49f-9167-48e7-bacd-91219748d5e1\") "
Nov 01 06:04:15.951109 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:15.951003    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9001d49f-9167-48e7-bacd-91219748d5e1-workload-certs" pod "9001d49f-9167-48e7-bacd-91219748d5e1" (UID: "9001d49f-9167-48e7-bacd-91219748d5e1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:04:15.951109 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:15.951009    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9001d49f-9167-48e7-bacd-91219748d5e1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9001d49f-9167-48e7-bacd-91219748d5e1-istio-data\") pod \"9001d49f-9167-48e7-bacd-91219748d5e1\" (UID: \"9001d49f-9167-48e7-bacd-91219748d5e1\") "
Nov 01 06:04:15.951109 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:15.951103    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea6d76c0-414c-4ae6-9c9a-01e41971592f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea6d76c0-414c-4ae6-9c9a-01e41971592f-workload-certs\") pod \"ea6d76c0-414c-4ae6-9c9a-01e41971592f\" (UID: \"ea6d76c0-414c-4ae6-9c9a-01e41971592f\") "
Nov 01 06:04:15.951674 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:15.951187    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea6d76c0-414c-4ae6-9c9a-01e41971592f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea6d76c0-414c-4ae6-9c9a-01e41971592f-workload-socket\") pod \"ea6d76c0-414c-4ae6-9c9a-01e41971592f\" (UID: \"ea6d76c0-414c-4ae6-9c9a-01e41971592f\") "
Nov 01 06:04:15.951674 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:15.951238    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9001d49f-9167-48e7-bacd-91219748d5e1-istio-token\" (UniqueName: \"kubernetes.io/projected/9001d49f-9167-48e7-bacd-91219748d5e1-istio-token\") pod \"9001d49f-9167-48e7-bacd-91219748d5e1\" (UID: \"9001d49f-9167-48e7-bacd-91219748d5e1\") "
Nov 01 06:04:15.951674 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:15.951269    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9001d49f-9167-48e7-bacd-91219748d5e1-kube-api-access-7cdgf\" (UniqueName: \"kubernetes.io/projected/9001d49f-9167-48e7-bacd-91219748d5e1-kube-api-access-7cdgf\") pod \"9001d49f-9167-48e7-bacd-91219748d5e1\" (UID: \"9001d49f-9167-48e7-bacd-91219748d5e1\") "
Nov 01 06:04:15.951674 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:15.951302    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea6d76c0-414c-4ae6-9c9a-01e41971592f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea6d76c0-414c-4ae6-9c9a-01e41971592f-istio-data\") pod \"ea6d76c0-414c-4ae6-9c9a-01e41971592f\" (UID: \"ea6d76c0-414c-4ae6-9c9a-01e41971592f\") "
Nov 01 06:04:15.951674 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:15.951339    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9001d49f-9167-48e7-bacd-91219748d5e1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9001d49f-9167-48e7-bacd-91219748d5e1-istiod-ca-cert\") pod \"9001d49f-9167-48e7-bacd-91219748d5e1\" (UID: \"9001d49f-9167-48e7-bacd-91219748d5e1\") "
Nov 01 06:04:15.951674 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:15.951375    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ea6d76c0-414c-4ae6-9c9a-01e41971592f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea6d76c0-414c-4ae6-9c9a-01e41971592f-istio-podinfo\") pod \"ea6d76c0-414c-4ae6-9c9a-01e41971592f\" (UID: \"ea6d76c0-414c-4ae6-9c9a-01e41971592f\") "
Nov 01 06:04:15.951674 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:15.951437    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea6d76c0-414c-4ae6-9c9a-01e41971592f-kube-api-access-6mpqq\" (UniqueName: \"kubernetes.io/projected/ea6d76c0-414c-4ae6-9c9a-01e41971592f-kube-api-access-6mpqq\") pod \"ea6d76c0-414c-4ae6-9c9a-01e41971592f\" (UID: \"ea6d76c0-414c-4ae6-9c9a-01e41971592f\") "
Nov 01 06:04:15.952076 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:15.952036    5689 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9001d49f-9167-48e7-bacd-91219748d5e1-workload-certs\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:04:15.952377 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:15.951187    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9001d49f-9167-48e7-bacd-91219748d5e1-istio-data" pod "9001d49f-9167-48e7-bacd-91219748d5e1" (UID: "9001d49f-9167-48e7-bacd-91219748d5e1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:04:15.952525 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:15.951220    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea6d76c0-414c-4ae6-9c9a-01e41971592f-istiod-ca-cert" pod "ea6d76c0-414c-4ae6-9c9a-01e41971592f" (UID: "ea6d76c0-414c-4ae6-9c9a-01e41971592f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:04:15.952525 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:15.951638    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9001d49f-9167-48e7-bacd-91219748d5e1-workload-socket" pod "9001d49f-9167-48e7-bacd-91219748d5e1" (UID: "9001d49f-9167-48e7-bacd-91219748d5e1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:04:15.952689 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:15.951875    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea6d76c0-414c-4ae6-9c9a-01e41971592f-workload-socket" pod "ea6d76c0-414c-4ae6-9c9a-01e41971592f" (UID: "ea6d76c0-414c-4ae6-9c9a-01e41971592f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:04:15.952689 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:15.952019    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea6d76c0-414c-4ae6-9c9a-01e41971592f-workload-certs" pod "ea6d76c0-414c-4ae6-9c9a-01e41971592f" (UID: "ea6d76c0-414c-4ae6-9c9a-01e41971592f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:04:15.952689 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:15.952169    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea6d76c0-414c-4ae6-9c9a-01e41971592f-istio-data" pod "ea6d76c0-414c-4ae6-9c9a-01e41971592f" (UID: "ea6d76c0-414c-4ae6-9c9a-01e41971592f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:04:15.952689 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:15.952342    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea6d76c0-414c-4ae6-9c9a-01e41971592f-credential-socket" pod "ea6d76c0-414c-4ae6-9c9a-01e41971592f" (UID: "ea6d76c0-414c-4ae6-9c9a-01e41971592f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:04:15.952972 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:15.952951    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9001d49f-9167-48e7-bacd-91219748d5e1-credential-socket" pod "9001d49f-9167-48e7-bacd-91219748d5e1" (UID: "9001d49f-9167-48e7-bacd-91219748d5e1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:04:15.953049 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:15.953021    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9001d49f-9167-48e7-bacd-91219748d5e1-istiod-ca-cert" pod "9001d49f-9167-48e7-bacd-91219748d5e1" (UID: "9001d49f-9167-48e7-bacd-91219748d5e1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:04:15.954194 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:15.954155    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9001d49f-9167-48e7-bacd-91219748d5e1-istio-envoy" pod "9001d49f-9167-48e7-bacd-91219748d5e1" (UID: "9001d49f-9167-48e7-bacd-91219748d5e1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:04:15.954702 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:15.954676    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9001d49f-9167-48e7-bacd-91219748d5e1-istio-podinfo" pod "9001d49f-9167-48e7-bacd-91219748d5e1" (UID: "9001d49f-9167-48e7-bacd-91219748d5e1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:04:15.955295 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:15.955258    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9001d49f-9167-48e7-bacd-91219748d5e1-kube-api-access-7cdgf" pod "9001d49f-9167-48e7-bacd-91219748d5e1" (UID: "9001d49f-9167-48e7-bacd-91219748d5e1"). InnerVolumeSpecName "kube-api-access-7cdgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:04:15.955532 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:15.955505    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea6d76c0-414c-4ae6-9c9a-01e41971592f-kube-api-access-6mpqq" pod "ea6d76c0-414c-4ae6-9c9a-01e41971592f" (UID: "ea6d76c0-414c-4ae6-9c9a-01e41971592f"). InnerVolumeSpecName "kube-api-access-6mpqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:04:15.955532 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:15.955511    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea6d76c0-414c-4ae6-9c9a-01e41971592f-istio-envoy" pod "ea6d76c0-414c-4ae6-9c9a-01e41971592f" (UID: "ea6d76c0-414c-4ae6-9c9a-01e41971592f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:04:15.956373 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:15.956346    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea6d76c0-414c-4ae6-9c9a-01e41971592f-istio-podinfo" pod "ea6d76c0-414c-4ae6-9c9a-01e41971592f" (UID: "ea6d76c0-414c-4ae6-9c9a-01e41971592f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:04:15.956605 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:15.956572    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea6d76c0-414c-4ae6-9c9a-01e41971592f-istio-token" pod "ea6d76c0-414c-4ae6-9c9a-01e41971592f" (UID: "ea6d76c0-414c-4ae6-9c9a-01e41971592f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:04:15.956769 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:15.956740    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9001d49f-9167-48e7-bacd-91219748d5e1-istio-token" pod "9001d49f-9167-48e7-bacd-91219748d5e1" (UID: "9001d49f-9167-48e7-bacd-91219748d5e1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:04:16.052963 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:16.052888    5689 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9001d49f-9167-48e7-bacd-91219748d5e1-workload-socket\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:04:16.052963 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:16.052912    5689 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9001d49f-9167-48e7-bacd-91219748d5e1-credential-socket\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:04:16.052963 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:16.052922    5689 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea6d76c0-414c-4ae6-9c9a-01e41971592f-istio-token\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:04:16.052963 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:16.052932    5689 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea6d76c0-414c-4ae6-9c9a-01e41971592f-istio-envoy\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:04:16.052963 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:16.052940    5689 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9001d49f-9167-48e7-bacd-91219748d5e1-istio-podinfo\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:04:16.052963 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:16.052950    5689 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9001d49f-9167-48e7-bacd-91219748d5e1-istio-data\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:04:16.052963 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:16.052958    5689 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea6d76c0-414c-4ae6-9c9a-01e41971592f-workload-certs\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:04:16.052963 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:16.052967    5689 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea6d76c0-414c-4ae6-9c9a-01e41971592f-workload-socket\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:04:16.053349 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:16.052976    5689 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9001d49f-9167-48e7-bacd-91219748d5e1-istio-token\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:04:16.053349 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:16.052988    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cdgf\" (UniqueName: \"kubernetes.io/projected/9001d49f-9167-48e7-bacd-91219748d5e1-kube-api-access-7cdgf\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:04:16.053349 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:16.052996    5689 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea6d76c0-414c-4ae6-9c9a-01e41971592f-istio-data\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:04:16.053349 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:16.053004    5689 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9001d49f-9167-48e7-bacd-91219748d5e1-istiod-ca-cert\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:04:16.053349 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:16.053012    5689 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea6d76c0-414c-4ae6-9c9a-01e41971592f-istio-podinfo\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:04:16.053349 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:16.053020    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mpqq\" (UniqueName: \"kubernetes.io/projected/ea6d76c0-414c-4ae6-9c9a-01e41971592f-kube-api-access-6mpqq\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:04:16.053349 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:16.053028    5689 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9001d49f-9167-48e7-bacd-91219748d5e1-istio-envoy\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:04:16.053349 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:16.053035    5689 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea6d76c0-414c-4ae6-9c9a-01e41971592f-istiod-ca-cert\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:04:16.053349 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:16.053044    5689 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea6d76c0-414c-4ae6-9c9a-01e41971592f-credential-socket\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:04:16.327309 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:16.327225    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:04:16.408550 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:16.408518    5689 generic.go:356] "Generic (PLEG): container finished" podID="9001d49f-9167-48e7-bacd-91219748d5e1" containerID="0062d22ba7f3f9526e542d6c2277e43627d76594563c663faa3e88bf44b73191" exitCode=0
Nov 01 06:04:16.408742 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:16.408608    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-fchpl"
Nov 01 06:04:16.408742 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:16.408622    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-fchpl" event={"ID":"9001d49f-9167-48e7-bacd-91219748d5e1","Type":"ContainerDied","Data":"0062d22ba7f3f9526e542d6c2277e43627d76594563c663faa3e88bf44b73191"}
Nov 01 06:04:16.408742 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:16.408689    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-fchpl" event={"ID":"9001d49f-9167-48e7-bacd-91219748d5e1","Type":"ContainerDied","Data":"c055d298a12200aec2d33e7cb33a97d90945f1239232924c74b82c353727a420"}
Nov 01 06:04:16.408742 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:16.408720    5689 scope.go:122] "RemoveContainer" containerID="0062d22ba7f3f9526e542d6c2277e43627d76594563c663faa3e88bf44b73191"
Nov 01 06:04:16.411753 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:16.411722    5689 generic.go:356] "Generic (PLEG): container finished" podID="ea6d76c0-414c-4ae6-9c9a-01e41971592f" containerID="c8321bede367ae90e95bf2919048be4282a992a0198f64bca75abbd5e02669cf" exitCode=0
Nov 01 06:04:16.411926 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:16.411784    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-5bms2" event={"ID":"ea6d76c0-414c-4ae6-9c9a-01e41971592f","Type":"ContainerDied","Data":"c8321bede367ae90e95bf2919048be4282a992a0198f64bca75abbd5e02669cf"}
Nov 01 06:04:16.411926 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:16.411818    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-5bms2"
Nov 01 06:04:16.411926 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:16.411839    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-5bms2" event={"ID":"ea6d76c0-414c-4ae6-9c9a-01e41971592f","Type":"ContainerDied","Data":"d2aa0f62f61253e21f11de49ec1c58950967de25c5705606a428312fc74d28c9"}
Nov 01 06:04:16.421790 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:16.421768    5689 scope.go:122] "RemoveContainer" containerID="0062d22ba7f3f9526e542d6c2277e43627d76594563c663faa3e88bf44b73191"
Nov 01 06:04:16.422101 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:04:16.422073    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0062d22ba7f3f9526e542d6c2277e43627d76594563c663faa3e88bf44b73191\": container with ID starting with 0062d22ba7f3f9526e542d6c2277e43627d76594563c663faa3e88bf44b73191 not found: ID does not exist" containerID="0062d22ba7f3f9526e542d6c2277e43627d76594563c663faa3e88bf44b73191"
Nov 01 06:04:16.422206 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:16.422113    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0062d22ba7f3f9526e542d6c2277e43627d76594563c663faa3e88bf44b73191"} err="failed to get container status \"0062d22ba7f3f9526e542d6c2277e43627d76594563c663faa3e88bf44b73191\": rpc error: code = NotFound desc = could not find container \"0062d22ba7f3f9526e542d6c2277e43627d76594563c663faa3e88bf44b73191\": container with ID starting with 0062d22ba7f3f9526e542d6c2277e43627d76594563c663faa3e88bf44b73191 not found: ID does not exist"
Nov 01 06:04:16.422206 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:16.422139    5689 scope.go:122] "RemoveContainer" containerID="c8321bede367ae90e95bf2919048be4282a992a0198f64bca75abbd5e02669cf"
Nov 01 06:04:16.431025 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:16.431007    5689 scope.go:122] "RemoveContainer" containerID="c8321bede367ae90e95bf2919048be4282a992a0198f64bca75abbd5e02669cf"
Nov 01 06:04:16.431339 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:04:16.431309    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8321bede367ae90e95bf2919048be4282a992a0198f64bca75abbd5e02669cf\": container with ID starting with c8321bede367ae90e95bf2919048be4282a992a0198f64bca75abbd5e02669cf not found: ID does not exist" containerID="c8321bede367ae90e95bf2919048be4282a992a0198f64bca75abbd5e02669cf"
Nov 01 06:04:16.431428 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:16.431348    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8321bede367ae90e95bf2919048be4282a992a0198f64bca75abbd5e02669cf"} err="failed to get container status \"c8321bede367ae90e95bf2919048be4282a992a0198f64bca75abbd5e02669cf\": rpc error: code = NotFound desc = could not find container \"c8321bede367ae90e95bf2919048be4282a992a0198f64bca75abbd5e02669cf\": container with ID starting with c8321bede367ae90e95bf2919048be4282a992a0198f64bca75abbd5e02669cf not found: ID does not exist"
Nov 01 06:04:16.572833 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:16.572799    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-fchpl"]
Nov 01 06:04:16.579682 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:16.579622    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-fchpl"]
Nov 01 06:04:16.693353 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:16.693317    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-5bms2"]
Nov 01 06:04:16.698171 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:16.698144    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-5bms2"]
Nov 01 06:04:18.067314 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:18.067283    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9001d49f-9167-48e7-bacd-91219748d5e1" path="/var/lib/kubelet/pods/9001d49f-9167-48e7-bacd-91219748d5e1/volumes"
Nov 01 06:04:18.068080 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:18.068058    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea6d76c0-414c-4ae6-9c9a-01e41971592f" path="/var/lib/kubelet/pods/ea6d76c0-414c-4ae6-9c9a-01e41971592f/volumes"
Nov 01 06:04:18.410197 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:18.410170    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:04:18.584091 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:18.584065    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:04:19.521786 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:19.521755    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:04:21.106286 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:21.106254    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:04:22.452501 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:22.452471    5689 generic.go:356] "Generic (PLEG): container finished" podID="3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2" containerID="7f8e1ce9f56c382921c26ab4727315b9bdcd2dafbc1fa1337085dbb0a20c4ab2" exitCode=0
Nov 01 06:04:22.452868 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:22.452531    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gk2pd" event={"ID":"3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2","Type":"ContainerDied","Data":"7f8e1ce9f56c382921c26ab4727315b9bdcd2dafbc1fa1337085dbb0a20c4ab2"}
Nov 01 06:04:22.526685 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:22.526659    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gk2pd"
Nov 01 06:04:22.618363 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:22.618329    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2-credential-socket\") pod \"3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2\" (UID: \"3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2\") "
Nov 01 06:04:22.618624 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:22.618400    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2-workload-socket\") pod \"3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2\" (UID: \"3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2\") "
Nov 01 06:04:22.618624 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:22.618432    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2-istio-data\") pod \"3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2\" (UID: \"3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2\") "
Nov 01 06:04:22.618624 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:22.618494    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2-istiod-ca-cert\") pod \"3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2\" (UID: \"3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2\") "
Nov 01 06:04:22.618624 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:22.618535    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2-istio-envoy\") pod \"3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2\" (UID: \"3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2\") "
Nov 01 06:04:22.618624 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:22.618597    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2-kube-api-access-p6429\" (UniqueName: \"kubernetes.io/projected/3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2-kube-api-access-p6429\") pod \"3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2\" (UID: \"3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2\") "
Nov 01 06:04:22.618624 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:22.618609    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2-credential-socket" pod "3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2" (UID: "3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:04:22.618941 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:22.618650    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2-workload-certs\") pod \"3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2\" (UID: \"3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2\") "
Nov 01 06:04:22.618941 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:22.618698    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2-workload-socket" pod "3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2" (UID: "3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:04:22.618941 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:22.618707    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2-istio-token\" (UniqueName: \"kubernetes.io/projected/3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2-istio-token\") pod \"3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2\" (UID: \"3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2\") "
Nov 01 06:04:22.618941 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:22.618740    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2-istio-data" pod "3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2" (UID: "3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:04:22.618941 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:22.618811    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2-istio-podinfo\") pod \"3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2\" (UID: \"3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2\") "
Nov 01 06:04:22.618941 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:22.618909    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2-istiod-ca-cert" pod "3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2" (UID: "3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:04:22.619253 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:22.619001    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2-workload-certs" pod "3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2" (UID: "3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:04:22.619513 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:22.619442    5689 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2-workload-socket\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:04:22.619513 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:22.619503    5689 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2-istio-data\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:04:22.619649 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:22.619518    5689 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2-istiod-ca-cert\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:04:22.619649 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:22.619532    5689 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2-workload-certs\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:04:22.619649 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:22.619547    5689 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2-credential-socket\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:04:22.620893 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:22.620862    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2-kube-api-access-p6429" pod "3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2" (UID: "3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2"). InnerVolumeSpecName "kube-api-access-p6429". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:04:22.620996 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:22.620872    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2-istio-podinfo" pod "3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2" (UID: "3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:04:22.620996 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:22.620896    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2-istio-envoy" pod "3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2" (UID: "3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:04:22.620996 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:22.620965    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2-istio-token" pod "3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2" (UID: "3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:04:22.720728 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:22.720699    5689 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2-istio-envoy\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:04:22.720728 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:22.720727    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6429\" (UniqueName: \"kubernetes.io/projected/3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2-kube-api-access-p6429\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:04:22.720920 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:22.720738    5689 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2-istio-token\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:04:22.720920 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:22.720747    5689 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2-istio-podinfo\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:04:23.460412 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:23.460378    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gk2pd"
Nov 01 06:04:23.460905 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:23.460372    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gk2pd" event={"ID":"3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2","Type":"ContainerDied","Data":"7545c6f44257cd1a7b6703e9605f55e620ce93f2bf69c32e97557011fb81264a"}
Nov 01 06:04:23.460905 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:23.460522    5689 scope.go:122] "RemoveContainer" containerID="7f8e1ce9f56c382921c26ab4727315b9bdcd2dafbc1fa1337085dbb0a20c4ab2"
Nov 01 06:04:23.672799 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:23.672771    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gk2pd"]
Nov 01 06:04:23.722158 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:23.722090    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-a48099-openshift-default-577864759-gk2pd"]
Nov 01 06:04:24.067733 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:24.067663    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2" path="/var/lib/kubelet/pods/3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2/volumes"
Nov 01 06:04:25.928310 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:25.928272    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:04:29.961235 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:29.961202    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1c4191-76f6bc474d-m8ljd"]
Nov 01 06:04:29.961773 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:29.961389    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1c4191-76f6bc474d-m8ljd" podUID="b9880a0a-c2b1-4a5e-a7b7-967038e13379" containerName="maas-api" containerID="cri-o://c88fc06982775064416be7d5a63ea00dfa4f6c024afec313f6df88024daeec37" gracePeriod=30
Nov 01 06:04:30.075234 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:04:30.075198    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 06:04:30.103800 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:30.103774    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:04:30.233845 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:30.233815    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1c4191-76f6bc474d-m8ljd"
Nov 01 06:04:30.290904 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:30.290877    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9880a0a-c2b1-4a5e-a7b7-967038e13379-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9880a0a-c2b1-4a5e-a7b7-967038e13379-openshift-service-ca\") pod \"b9880a0a-c2b1-4a5e-a7b7-967038e13379\" (UID: \"b9880a0a-c2b1-4a5e-a7b7-967038e13379\") "
Nov 01 06:04:30.291051 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:30.290948    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9880a0a-c2b1-4a5e-a7b7-967038e13379-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9880a0a-c2b1-4a5e-a7b7-967038e13379-maas-api-tls\") pod \"b9880a0a-c2b1-4a5e-a7b7-967038e13379\" (UID: \"b9880a0a-c2b1-4a5e-a7b7-967038e13379\") "
Nov 01 06:04:30.291051 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:30.290994    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9880a0a-c2b1-4a5e-a7b7-967038e13379-kube-api-access-96pxn\" (UniqueName: \"kubernetes.io/projected/b9880a0a-c2b1-4a5e-a7b7-967038e13379-kube-api-access-96pxn\") pod \"b9880a0a-c2b1-4a5e-a7b7-967038e13379\" (UID: \"b9880a0a-c2b1-4a5e-a7b7-967038e13379\") "
Nov 01 06:04:30.291051 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:30.291024    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9880a0a-c2b1-4a5e-a7b7-967038e13379-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9880a0a-c2b1-4a5e-a7b7-967038e13379-metrics-certs\") pod \"b9880a0a-c2b1-4a5e-a7b7-967038e13379\" (UID: \"b9880a0a-c2b1-4a5e-a7b7-967038e13379\") "
Nov 01 06:04:30.291310 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:30.291281    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9880a0a-c2b1-4a5e-a7b7-967038e13379-openshift-service-ca" pod "b9880a0a-c2b1-4a5e-a7b7-967038e13379" (UID: "b9880a0a-c2b1-4a5e-a7b7-967038e13379"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:04:30.293230 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:30.293209    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9880a0a-c2b1-4a5e-a7b7-967038e13379-kube-api-access-96pxn" pod "b9880a0a-c2b1-4a5e-a7b7-967038e13379" (UID: "b9880a0a-c2b1-4a5e-a7b7-967038e13379"). InnerVolumeSpecName "kube-api-access-96pxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:04:30.293230 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:30.293211    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9880a0a-c2b1-4a5e-a7b7-967038e13379-metrics-certs" pod "b9880a0a-c2b1-4a5e-a7b7-967038e13379" (UID: "b9880a0a-c2b1-4a5e-a7b7-967038e13379"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:04:30.293338 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:30.293287    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9880a0a-c2b1-4a5e-a7b7-967038e13379-maas-api-tls" pod "b9880a0a-c2b1-4a5e-a7b7-967038e13379" (UID: "b9880a0a-c2b1-4a5e-a7b7-967038e13379"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:04:30.392129 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:30.392090    5689 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9880a0a-c2b1-4a5e-a7b7-967038e13379-openshift-service-ca\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:04:30.392129 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:30.392122    5689 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9880a0a-c2b1-4a5e-a7b7-967038e13379-maas-api-tls\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:04:30.392129 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:30.392131    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96pxn\" (UniqueName: \"kubernetes.io/projected/b9880a0a-c2b1-4a5e-a7b7-967038e13379-kube-api-access-96pxn\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:04:30.392360 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:30.392141    5689 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9880a0a-c2b1-4a5e-a7b7-967038e13379-metrics-certs\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:04:30.498045 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:30.497949    5689 generic.go:356] "Generic (PLEG): container finished" podID="b9880a0a-c2b1-4a5e-a7b7-967038e13379" containerID="c88fc06982775064416be7d5a63ea00dfa4f6c024afec313f6df88024daeec37" exitCode=0
Nov 01 06:04:30.498045 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:30.498029    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1c4191-76f6bc474d-m8ljd"
Nov 01 06:04:30.498316 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:30.498049    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1c4191-76f6bc474d-m8ljd" event={"ID":"b9880a0a-c2b1-4a5e-a7b7-967038e13379","Type":"ContainerDied","Data":"c88fc06982775064416be7d5a63ea00dfa4f6c024afec313f6df88024daeec37"}
Nov 01 06:04:30.498316 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:30.498106    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1c4191-76f6bc474d-m8ljd" event={"ID":"b9880a0a-c2b1-4a5e-a7b7-967038e13379","Type":"ContainerDied","Data":"936a6a78876940c8dc01cd8c12661ed4720493de4ceb2fc3f15677fa79695c1d"}
Nov 01 06:04:30.498316 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:30.498137    5689 scope.go:122] "RemoveContainer" containerID="c88fc06982775064416be7d5a63ea00dfa4f6c024afec313f6df88024daeec37"
Nov 01 06:04:30.508903 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:30.508887    5689 scope.go:122] "RemoveContainer" containerID="c88fc06982775064416be7d5a63ea00dfa4f6c024afec313f6df88024daeec37"
Nov 01 06:04:30.509150 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:04:30.509131    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c88fc06982775064416be7d5a63ea00dfa4f6c024afec313f6df88024daeec37\": container with ID starting with c88fc06982775064416be7d5a63ea00dfa4f6c024afec313f6df88024daeec37 not found: ID does not exist" containerID="c88fc06982775064416be7d5a63ea00dfa4f6c024afec313f6df88024daeec37"
Nov 01 06:04:30.509201 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:30.509155    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c88fc06982775064416be7d5a63ea00dfa4f6c024afec313f6df88024daeec37"} err="failed to get container status \"c88fc06982775064416be7d5a63ea00dfa4f6c024afec313f6df88024daeec37\": rpc error: code = NotFound desc = could not find container \"c88fc06982775064416be7d5a63ea00dfa4f6c024afec313f6df88024daeec37\": container with ID starting with c88fc06982775064416be7d5a63ea00dfa4f6c024afec313f6df88024daeec37 not found: ID does not exist"
Nov 01 06:04:30.660254 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:30.660223    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1c4191-76f6bc474d-m8ljd"]
Nov 01 06:04:30.664492 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:30.664468    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1c4191-76f6bc474d-m8ljd"]
Nov 01 06:04:31.817143 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:31.817114    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:04:32.073901 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:32.073814    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9880a0a-c2b1-4a5e-a7b7-967038e13379" path="/var/lib/kubelet/pods/b9880a0a-c2b1-4a5e-a7b7-967038e13379/volumes"
Nov 01 06:04:36.694940 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:36.694906    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:04:37.240155 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:37.240124    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:04:37.425441 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:37.425406    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:04:38.844558 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:38.844517    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:04:39.290050 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:39.290018    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm"]
Nov 01 06:04:39.290286 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:39.290266    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm" podUID="b7183c3f-d319-4c01-83fe-0df57b41b69c" containerName="istio-proxy" containerID="cri-o://56283a243135e0b01e7c68283761ff9fa9d93a8b4537d04459a2df5c0605b3ec" gracePeriod=30
Nov 01 06:04:39.328181 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:39.328149    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:04:39.763244 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:39.763215    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:04:41.929049 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:41.929021    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:04:44.535581 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:44.535554    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm"
Nov 01 06:04:44.569794 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:44.569736    5689 generic.go:356] "Generic (PLEG): container finished" podID="b7183c3f-d319-4c01-83fe-0df57b41b69c" containerID="56283a243135e0b01e7c68283761ff9fa9d93a8b4537d04459a2df5c0605b3ec" exitCode=0
Nov 01 06:04:44.569794 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:44.569783    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm"
Nov 01 06:04:44.569932 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:44.569809    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm" event={"ID":"b7183c3f-d319-4c01-83fe-0df57b41b69c","Type":"ContainerDied","Data":"56283a243135e0b01e7c68283761ff9fa9d93a8b4537d04459a2df5c0605b3ec"}
Nov 01 06:04:44.569932 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:44.569854    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm" event={"ID":"b7183c3f-d319-4c01-83fe-0df57b41b69c","Type":"ContainerDied","Data":"f0e66f4e70220eb59dc5240d97c7314fb57fdd285328541e825d92702a0b4527"}
Nov 01 06:04:44.569932 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:44.569876    5689 scope.go:122] "RemoveContainer" containerID="56283a243135e0b01e7c68283761ff9fa9d93a8b4537d04459a2df5c0605b3ec"
Nov 01 06:04:44.578689 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:44.578671    5689 scope.go:122] "RemoveContainer" containerID="f6f04e2a6248b9e4b0e00812f8807e1ec4d084072d7bdb2481a48e70de33447a"
Nov 01 06:04:44.586699 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:44.586684    5689 scope.go:122] "RemoveContainer" containerID="56283a243135e0b01e7c68283761ff9fa9d93a8b4537d04459a2df5c0605b3ec"
Nov 01 06:04:44.586942 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:04:44.586926    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56283a243135e0b01e7c68283761ff9fa9d93a8b4537d04459a2df5c0605b3ec\": container with ID starting with 56283a243135e0b01e7c68283761ff9fa9d93a8b4537d04459a2df5c0605b3ec not found: ID does not exist" containerID="56283a243135e0b01e7c68283761ff9fa9d93a8b4537d04459a2df5c0605b3ec"
Nov 01 06:04:44.586987 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:44.586951    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56283a243135e0b01e7c68283761ff9fa9d93a8b4537d04459a2df5c0605b3ec"} err="failed to get container status \"56283a243135e0b01e7c68283761ff9fa9d93a8b4537d04459a2df5c0605b3ec\": rpc error: code = NotFound desc = could not find container \"56283a243135e0b01e7c68283761ff9fa9d93a8b4537d04459a2df5c0605b3ec\": container with ID starting with 56283a243135e0b01e7c68283761ff9fa9d93a8b4537d04459a2df5c0605b3ec not found: ID does not exist"
Nov 01 06:04:44.586987 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:44.586972    5689 scope.go:122] "RemoveContainer" containerID="f6f04e2a6248b9e4b0e00812f8807e1ec4d084072d7bdb2481a48e70de33447a"
Nov 01 06:04:44.587203 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:04:44.587188    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6f04e2a6248b9e4b0e00812f8807e1ec4d084072d7bdb2481a48e70de33447a\": container with ID starting with f6f04e2a6248b9e4b0e00812f8807e1ec4d084072d7bdb2481a48e70de33447a not found: ID does not exist" containerID="f6f04e2a6248b9e4b0e00812f8807e1ec4d084072d7bdb2481a48e70de33447a"
Nov 01 06:04:44.587244 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:44.587207    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6f04e2a6248b9e4b0e00812f8807e1ec4d084072d7bdb2481a48e70de33447a"} err="failed to get container status \"f6f04e2a6248b9e4b0e00812f8807e1ec4d084072d7bdb2481a48e70de33447a\": rpc error: code = NotFound desc = could not find container \"f6f04e2a6248b9e4b0e00812f8807e1ec4d084072d7bdb2481a48e70de33447a\": container with ID starting with f6f04e2a6248b9e4b0e00812f8807e1ec4d084072d7bdb2481a48e70de33447a not found: ID does not exist"
Nov 01 06:04:44.632948 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:44.632928    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7183c3f-d319-4c01-83fe-0df57b41b69c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7183c3f-d319-4c01-83fe-0df57b41b69c-istio-envoy\") pod \"b7183c3f-d319-4c01-83fe-0df57b41b69c\" (UID: \"b7183c3f-d319-4c01-83fe-0df57b41b69c\") "
Nov 01 06:04:44.633034 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:44.632987    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7183c3f-d319-4c01-83fe-0df57b41b69c-kube-api-access-8q282\" (UniqueName: \"kubernetes.io/projected/b7183c3f-d319-4c01-83fe-0df57b41b69c-kube-api-access-8q282\") pod \"b7183c3f-d319-4c01-83fe-0df57b41b69c\" (UID: \"b7183c3f-d319-4c01-83fe-0df57b41b69c\") "
Nov 01 06:04:44.633034 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:44.633022    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7183c3f-d319-4c01-83fe-0df57b41b69c-istio-token\" (UniqueName: \"kubernetes.io/projected/b7183c3f-d319-4c01-83fe-0df57b41b69c-istio-token\") pod \"b7183c3f-d319-4c01-83fe-0df57b41b69c\" (UID: \"b7183c3f-d319-4c01-83fe-0df57b41b69c\") "
Nov 01 06:04:44.633147 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:44.633041    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7183c3f-d319-4c01-83fe-0df57b41b69c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7183c3f-d319-4c01-83fe-0df57b41b69c-credential-socket\") pod \"b7183c3f-d319-4c01-83fe-0df57b41b69c\" (UID: \"b7183c3f-d319-4c01-83fe-0df57b41b69c\") "
Nov 01 06:04:44.633147 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:44.633092    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b7183c3f-d319-4c01-83fe-0df57b41b69c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7183c3f-d319-4c01-83fe-0df57b41b69c-istio-podinfo\") pod \"b7183c3f-d319-4c01-83fe-0df57b41b69c\" (UID: \"b7183c3f-d319-4c01-83fe-0df57b41b69c\") "
Nov 01 06:04:44.633252 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:44.633212    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7183c3f-d319-4c01-83fe-0df57b41b69c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7183c3f-d319-4c01-83fe-0df57b41b69c-istiod-ca-cert\") pod \"b7183c3f-d319-4c01-83fe-0df57b41b69c\" (UID: \"b7183c3f-d319-4c01-83fe-0df57b41b69c\") "
Nov 01 06:04:44.633252 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:44.633246    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7183c3f-d319-4c01-83fe-0df57b41b69c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7183c3f-d319-4c01-83fe-0df57b41b69c-istio-data\") pod \"b7183c3f-d319-4c01-83fe-0df57b41b69c\" (UID: \"b7183c3f-d319-4c01-83fe-0df57b41b69c\") "
Nov 01 06:04:44.633366 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:44.633298    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7183c3f-d319-4c01-83fe-0df57b41b69c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7183c3f-d319-4c01-83fe-0df57b41b69c-workload-socket\") pod \"b7183c3f-d319-4c01-83fe-0df57b41b69c\" (UID: \"b7183c3f-d319-4c01-83fe-0df57b41b69c\") "
Nov 01 06:04:44.633366 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:44.633333    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7183c3f-d319-4c01-83fe-0df57b41b69c-credential-socket" pod "b7183c3f-d319-4c01-83fe-0df57b41b69c" (UID: "b7183c3f-d319-4c01-83fe-0df57b41b69c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:04:44.633519 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:44.633382    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7183c3f-d319-4c01-83fe-0df57b41b69c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7183c3f-d319-4c01-83fe-0df57b41b69c-workload-certs\") pod \"b7183c3f-d319-4c01-83fe-0df57b41b69c\" (UID: \"b7183c3f-d319-4c01-83fe-0df57b41b69c\") "
Nov 01 06:04:44.633587 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:44.633521    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7183c3f-d319-4c01-83fe-0df57b41b69c-istio-data" pod "b7183c3f-d319-4c01-83fe-0df57b41b69c" (UID: "b7183c3f-d319-4c01-83fe-0df57b41b69c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:04:44.633685 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:44.633663    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7183c3f-d319-4c01-83fe-0df57b41b69c-workload-socket" pod "b7183c3f-d319-4c01-83fe-0df57b41b69c" (UID: "b7183c3f-d319-4c01-83fe-0df57b41b69c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:04:44.633808 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:44.633755    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7183c3f-d319-4c01-83fe-0df57b41b69c-istiod-ca-cert" pod "b7183c3f-d319-4c01-83fe-0df57b41b69c" (UID: "b7183c3f-d319-4c01-83fe-0df57b41b69c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:04:44.634040 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:44.633857    5689 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7183c3f-d319-4c01-83fe-0df57b41b69c-istiod-ca-cert\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:04:44.634040 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:44.633871    5689 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7183c3f-d319-4c01-83fe-0df57b41b69c-istio-data\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:04:44.634040 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:44.633880    5689 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7183c3f-d319-4c01-83fe-0df57b41b69c-workload-socket\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:04:44.634040 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:44.633894    5689 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7183c3f-d319-4c01-83fe-0df57b41b69c-credential-socket\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:04:44.634040 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:44.634031    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7183c3f-d319-4c01-83fe-0df57b41b69c-workload-certs" pod "b7183c3f-d319-4c01-83fe-0df57b41b69c" (UID: "b7183c3f-d319-4c01-83fe-0df57b41b69c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:04:44.635287 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:44.635266    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b7183c3f-d319-4c01-83fe-0df57b41b69c-istio-podinfo" pod "b7183c3f-d319-4c01-83fe-0df57b41b69c" (UID: "b7183c3f-d319-4c01-83fe-0df57b41b69c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:04:44.635385 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:44.635373    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7183c3f-d319-4c01-83fe-0df57b41b69c-istio-token" pod "b7183c3f-d319-4c01-83fe-0df57b41b69c" (UID: "b7183c3f-d319-4c01-83fe-0df57b41b69c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:04:44.635694 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:44.635670    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7183c3f-d319-4c01-83fe-0df57b41b69c-istio-envoy" pod "b7183c3f-d319-4c01-83fe-0df57b41b69c" (UID: "b7183c3f-d319-4c01-83fe-0df57b41b69c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:04:44.635694 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:44.635684    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7183c3f-d319-4c01-83fe-0df57b41b69c-kube-api-access-8q282" pod "b7183c3f-d319-4c01-83fe-0df57b41b69c" (UID: "b7183c3f-d319-4c01-83fe-0df57b41b69c"). InnerVolumeSpecName "kube-api-access-8q282". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:04:44.698840 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:44.698813    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7d1ec3/distinct-w0-7d1ec3-kserve-58cdfc8d58-n7xpt"]
Nov 01 06:04:44.734733 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:44.734709    5689 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7183c3f-d319-4c01-83fe-0df57b41b69c-workload-certs\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:04:44.734733 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:44.734731    5689 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7183c3f-d319-4c01-83fe-0df57b41b69c-istio-envoy\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:04:44.734733 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:44.734742    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8q282\" (UniqueName: \"kubernetes.io/projected/b7183c3f-d319-4c01-83fe-0df57b41b69c-kube-api-access-8q282\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:04:44.734733 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:44.734751    5689 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7183c3f-d319-4c01-83fe-0df57b41b69c-istio-token\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:04:44.734988 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:44.734760    5689 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7183c3f-d319-4c01-83fe-0df57b41b69c-istio-podinfo\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:04:44.889389 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:44.889362    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:04:45.026737 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:45.026707    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm"]
Nov 01 06:04:45.030546 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:45.030519    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-1c4191-openshift-default-7d749f944c-9fsfm"]
Nov 01 06:04:45.899489 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:45.899436    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:04:46.066674 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:46.066644    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7183c3f-d319-4c01-83fe-0df57b41b69c" path="/var/lib/kubelet/pods/b7183c3f-d319-4c01-83fe-0df57b41b69c/volumes"
Nov 01 06:04:46.778265 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:46.778234    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:04:46.951877 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:46.951815    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:04:51.636334 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:51.636292    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:04:53.066564 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:53.066532    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:04:53.815402 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:53.815374    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:04:53.986016 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:53.985986    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:04:56.846206 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:56.846172    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:04:58.999152 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:58.999122    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:04:59.719156 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:59.719120    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-7d1ec3/distinct-w0-7d1ec3-kserve-58cdfc8d58-n7xpt" podUID="4a1fcaaf-0b04-4602-8985-866f6152303d" containerName="main" containerID="cri-o://231e59b47cff2533f6dc91b77dd4158b636975474c6867bbae7cf753f168693c" gracePeriod=45
Nov 01 06:04:59.970925 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:59.970869    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7d1ec3/distinct-w0-7d1ec3-kserve-58cdfc8d58-n7xpt"
Nov 01 06:04:59.989595 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:59.989576    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a1fcaaf-0b04-4602-8985-866f6152303d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a1fcaaf-0b04-4602-8985-866f6152303d-tmp-dir\") pod \"4a1fcaaf-0b04-4602-8985-866f6152303d\" (UID: \"4a1fcaaf-0b04-4602-8985-866f6152303d\") "
Nov 01 06:04:59.989691 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:59.989627    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a1fcaaf-0b04-4602-8985-866f6152303d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a1fcaaf-0b04-4602-8985-866f6152303d-dshm\") pod \"4a1fcaaf-0b04-4602-8985-866f6152303d\" (UID: \"4a1fcaaf-0b04-4602-8985-866f6152303d\") "
Nov 01 06:04:59.989691 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:59.989673    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a1fcaaf-0b04-4602-8985-866f6152303d-home\" (UniqueName: \"kubernetes.io/empty-dir/4a1fcaaf-0b04-4602-8985-866f6152303d-home\") pod \"4a1fcaaf-0b04-4602-8985-866f6152303d\" (UID: \"4a1fcaaf-0b04-4602-8985-866f6152303d\") "
Nov 01 06:04:59.989770 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:59.989704    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a1fcaaf-0b04-4602-8985-866f6152303d-kube-api-access-j4tp7\" (UniqueName: \"kubernetes.io/projected/4a1fcaaf-0b04-4602-8985-866f6152303d-kube-api-access-j4tp7\") pod \"4a1fcaaf-0b04-4602-8985-866f6152303d\" (UID: \"4a1fcaaf-0b04-4602-8985-866f6152303d\") "
Nov 01 06:04:59.989770 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:59.989752    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a1fcaaf-0b04-4602-8985-866f6152303d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a1fcaaf-0b04-4602-8985-866f6152303d-model-cache\") pod \"4a1fcaaf-0b04-4602-8985-866f6152303d\" (UID: \"4a1fcaaf-0b04-4602-8985-866f6152303d\") "
Nov 01 06:04:59.989868 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:59.989781    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a1fcaaf-0b04-4602-8985-866f6152303d-tls-certs\" (UniqueName: \"kubernetes.io/secret/4a1fcaaf-0b04-4602-8985-866f6152303d-tls-certs\") pod \"4a1fcaaf-0b04-4602-8985-866f6152303d\" (UID: \"4a1fcaaf-0b04-4602-8985-866f6152303d\") "
Nov 01 06:04:59.989927 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:59.989841    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a1fcaaf-0b04-4602-8985-866f6152303d-tmp-dir" pod "4a1fcaaf-0b04-4602-8985-866f6152303d" (UID: "4a1fcaaf-0b04-4602-8985-866f6152303d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:04:59.989927 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:59.989906    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a1fcaaf-0b04-4602-8985-866f6152303d-home" pod "4a1fcaaf-0b04-4602-8985-866f6152303d" (UID: "4a1fcaaf-0b04-4602-8985-866f6152303d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:04:59.990088 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:59.990066    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a1fcaaf-0b04-4602-8985-866f6152303d-model-cache" pod "4a1fcaaf-0b04-4602-8985-866f6152303d" (UID: "4a1fcaaf-0b04-4602-8985-866f6152303d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:04:59.990167 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:59.990137    5689 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a1fcaaf-0b04-4602-8985-866f6152303d-tmp-dir\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:04:59.990167 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:59.990153    5689 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a1fcaaf-0b04-4602-8985-866f6152303d-home\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:04:59.990271 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:59.990170    5689 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a1fcaaf-0b04-4602-8985-866f6152303d-model-cache\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:04:59.991860 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:59.991837    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a1fcaaf-0b04-4602-8985-866f6152303d-dshm" pod "4a1fcaaf-0b04-4602-8985-866f6152303d" (UID: "4a1fcaaf-0b04-4602-8985-866f6152303d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:04:59.991983 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:59.991968    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a1fcaaf-0b04-4602-8985-866f6152303d-tls-certs" pod "4a1fcaaf-0b04-4602-8985-866f6152303d" (UID: "4a1fcaaf-0b04-4602-8985-866f6152303d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:04:59.992032 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:04:59.992012    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a1fcaaf-0b04-4602-8985-866f6152303d-kube-api-access-j4tp7" pod "4a1fcaaf-0b04-4602-8985-866f6152303d" (UID: "4a1fcaaf-0b04-4602-8985-866f6152303d"). InnerVolumeSpecName "kube-api-access-j4tp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:05:00.068842 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:05:00.068810    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 06:05:00.091573 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:00.091533    5689 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a1fcaaf-0b04-4602-8985-866f6152303d-dshm\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:05:00.091573 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:00.091564    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4tp7\" (UniqueName: \"kubernetes.io/projected/4a1fcaaf-0b04-4602-8985-866f6152303d-kube-api-access-j4tp7\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:05:00.091770 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:00.091579    5689 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a1fcaaf-0b04-4602-8985-866f6152303d-tls-certs\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:05:00.646442 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:00.646409    5689 generic.go:356] "Generic (PLEG): container finished" podID="4a1fcaaf-0b04-4602-8985-866f6152303d" containerID="231e59b47cff2533f6dc91b77dd4158b636975474c6867bbae7cf753f168693c" exitCode=0
Nov 01 06:05:00.646629 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:00.646463    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7d1ec3/distinct-w0-7d1ec3-kserve-58cdfc8d58-n7xpt"
Nov 01 06:05:00.646629 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:00.646494    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7d1ec3/distinct-w0-7d1ec3-kserve-58cdfc8d58-n7xpt" event={"ID":"4a1fcaaf-0b04-4602-8985-866f6152303d","Type":"ContainerDied","Data":"231e59b47cff2533f6dc91b77dd4158b636975474c6867bbae7cf753f168693c"}
Nov 01 06:05:00.646629 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:00.646539    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7d1ec3/distinct-w0-7d1ec3-kserve-58cdfc8d58-n7xpt" event={"ID":"4a1fcaaf-0b04-4602-8985-866f6152303d","Type":"ContainerDied","Data":"8f762389e9cc8fea55cf391276ba50acb1ea3b57a034843ce937025b5b494cd6"}
Nov 01 06:05:00.646629 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:00.646563    5689 scope.go:122] "RemoveContainer" containerID="231e59b47cff2533f6dc91b77dd4158b636975474c6867bbae7cf753f168693c"
Nov 01 06:05:00.655299 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:00.655279    5689 scope.go:122] "RemoveContainer" containerID="231e59b47cff2533f6dc91b77dd4158b636975474c6867bbae7cf753f168693c"
Nov 01 06:05:00.655634 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:05:00.655615    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"231e59b47cff2533f6dc91b77dd4158b636975474c6867bbae7cf753f168693c\": container with ID starting with 231e59b47cff2533f6dc91b77dd4158b636975474c6867bbae7cf753f168693c not found: ID does not exist" containerID="231e59b47cff2533f6dc91b77dd4158b636975474c6867bbae7cf753f168693c"
Nov 01 06:05:00.655691 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:00.655645    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"231e59b47cff2533f6dc91b77dd4158b636975474c6867bbae7cf753f168693c"} err="failed to get container status \"231e59b47cff2533f6dc91b77dd4158b636975474c6867bbae7cf753f168693c\": rpc error: code = NotFound desc = could not find container \"231e59b47cff2533f6dc91b77dd4158b636975474c6867bbae7cf753f168693c\": container with ID starting with 231e59b47cff2533f6dc91b77dd4158b636975474c6867bbae7cf753f168693c not found: ID does not exist"
Nov 01 06:05:00.808736 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:00.808706    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7d1ec3/distinct-w0-7d1ec3-kserve-58cdfc8d58-n7xpt"]
Nov 01 06:05:00.858141 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:00.858113    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-7d1ec3/distinct-w0-7d1ec3-kserve-58cdfc8d58-n7xpt"]
Nov 01 06:05:00.945853 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:00.945818    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7d1ec3/distinct-w0-7d1ec3-kserve-58cdfc8d58-n7xpt" podUID="4a1fcaaf-0b04-4602-8985-866f6152303d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded"
Nov 01 06:05:01.619160 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:01.619127    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:05:02.067233 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:02.067197    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a1fcaaf-0b04-4602-8985-866f6152303d" path="/var/lib/kubelet/pods/4a1fcaaf-0b04-4602-8985-866f6152303d/volumes"
Nov 01 06:05:02.800245 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:02.800211    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:05:04.784025 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:04.783999    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:05:06.413220 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:06.413186    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:05:09.790439 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:09.790406    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:05:10.474774 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:10.474739    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-7d1ec3-55d75cf7c8-25l2f"]
Nov 01 06:05:10.474960 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:10.474936    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7d1ec3-55d75cf7c8-25l2f" podUID="f4ef14f8-4441-4dba-ace6-fad77c35a0b2" containerName="maas-api" containerID="cri-o://49dcfcca50502f54b8c1c41745057b970b73ef353cd43b4b642a4ae9cce2781a" gracePeriod=30
Nov 01 06:05:10.610353 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:10.610327    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:05:10.706195 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:10.706154    5689 generic.go:356] "Generic (PLEG): container finished" podID="f4ef14f8-4441-4dba-ace6-fad77c35a0b2" containerID="49dcfcca50502f54b8c1c41745057b970b73ef353cd43b4b642a4ae9cce2781a" exitCode=0
Nov 01 06:05:10.706361 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:10.706224    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7d1ec3-55d75cf7c8-25l2f" event={"ID":"f4ef14f8-4441-4dba-ace6-fad77c35a0b2","Type":"ContainerDied","Data":"49dcfcca50502f54b8c1c41745057b970b73ef353cd43b4b642a4ae9cce2781a"}
Nov 01 06:05:10.748849 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:10.748828    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7d1ec3-55d75cf7c8-25l2f"
Nov 01 06:05:10.803280 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:10.803256    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4ef14f8-4441-4dba-ace6-fad77c35a0b2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4ef14f8-4441-4dba-ace6-fad77c35a0b2-maas-api-tls\") pod \"f4ef14f8-4441-4dba-ace6-fad77c35a0b2\" (UID: \"f4ef14f8-4441-4dba-ace6-fad77c35a0b2\") "
Nov 01 06:05:10.803628 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:10.803364    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4ef14f8-4441-4dba-ace6-fad77c35a0b2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4ef14f8-4441-4dba-ace6-fad77c35a0b2-openshift-service-ca\") pod \"f4ef14f8-4441-4dba-ace6-fad77c35a0b2\" (UID: \"f4ef14f8-4441-4dba-ace6-fad77c35a0b2\") "
Nov 01 06:05:10.803628 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:10.803397    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4ef14f8-4441-4dba-ace6-fad77c35a0b2-kube-api-access-729lb\" (UniqueName: \"kubernetes.io/projected/f4ef14f8-4441-4dba-ace6-fad77c35a0b2-kube-api-access-729lb\") pod \"f4ef14f8-4441-4dba-ace6-fad77c35a0b2\" (UID: \"f4ef14f8-4441-4dba-ace6-fad77c35a0b2\") "
Nov 01 06:05:10.803628 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:10.803505    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4ef14f8-4441-4dba-ace6-fad77c35a0b2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4ef14f8-4441-4dba-ace6-fad77c35a0b2-metrics-certs\") pod \"f4ef14f8-4441-4dba-ace6-fad77c35a0b2\" (UID: \"f4ef14f8-4441-4dba-ace6-fad77c35a0b2\") "
Nov 01 06:05:10.803750 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:10.803652    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4ef14f8-4441-4dba-ace6-fad77c35a0b2-openshift-service-ca" pod "f4ef14f8-4441-4dba-ace6-fad77c35a0b2" (UID: "f4ef14f8-4441-4dba-ace6-fad77c35a0b2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:05:10.803900 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:10.803881    5689 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4ef14f8-4441-4dba-ace6-fad77c35a0b2-openshift-service-ca\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:05:10.805588 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:10.805564    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4ef14f8-4441-4dba-ace6-fad77c35a0b2-kube-api-access-729lb" pod "f4ef14f8-4441-4dba-ace6-fad77c35a0b2" (UID: "f4ef14f8-4441-4dba-ace6-fad77c35a0b2"). InnerVolumeSpecName "kube-api-access-729lb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:05:10.805704 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:10.805635    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4ef14f8-4441-4dba-ace6-fad77c35a0b2-maas-api-tls" pod "f4ef14f8-4441-4dba-ace6-fad77c35a0b2" (UID: "f4ef14f8-4441-4dba-ace6-fad77c35a0b2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:05:10.805704 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:10.805652    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4ef14f8-4441-4dba-ace6-fad77c35a0b2-metrics-certs" pod "f4ef14f8-4441-4dba-ace6-fad77c35a0b2" (UID: "f4ef14f8-4441-4dba-ace6-fad77c35a0b2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:05:10.904436 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:10.904405    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-729lb\" (UniqueName: \"kubernetes.io/projected/f4ef14f8-4441-4dba-ace6-fad77c35a0b2-kube-api-access-729lb\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:05:10.904436 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:10.904432    5689 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4ef14f8-4441-4dba-ace6-fad77c35a0b2-metrics-certs\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:05:10.904436 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:10.904442    5689 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4ef14f8-4441-4dba-ace6-fad77c35a0b2-maas-api-tls\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:05:11.712711 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:11.712678    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7d1ec3-55d75cf7c8-25l2f"
Nov 01 06:05:11.712938 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:11.712709    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7d1ec3-55d75cf7c8-25l2f" event={"ID":"f4ef14f8-4441-4dba-ace6-fad77c35a0b2","Type":"ContainerDied","Data":"fdbe47df34c6d88b76283f2a774ab8f5eeb7be693506bf11c4a01840b9934542"}
Nov 01 06:05:11.712938 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:11.712771    5689 scope.go:122] "RemoveContainer" containerID="49dcfcca50502f54b8c1c41745057b970b73ef353cd43b4b642a4ae9cce2781a"
Nov 01 06:05:11.821895 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:11.821853    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:05:11.875842 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:11.875811    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-7d1ec3-55d75cf7c8-25l2f"]
Nov 01 06:05:11.924684 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:11.924661    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-7d1ec3-55d75cf7c8-25l2f"]
Nov 01 06:05:11.997322 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:11.997244    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:05:12.066507 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:12.066478    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4ef14f8-4441-4dba-ace6-fad77c35a0b2" path="/var/lib/kubelet/pods/f4ef14f8-4441-4dba-ace6-fad77c35a0b2/volumes"
Nov 01 06:05:12.423046 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:12.422998    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:05:14.726677 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:14.726622    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:05:17.276907 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:17.276779    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:05:19.465940 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:19.465912    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:05:19.650191 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:19.650155    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:05:22.006134 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:22.006054    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-9npk6"]
Nov 01 06:05:22.006582 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:22.006535    5689 cpu_manager.go:414] "containerMap: removing container" podUID="b7183c3f-d319-4c01-83fe-0df57b41b69c" containerName="istio-proxy"
Nov 01 06:05:22.006582 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:22.006550    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7183c3f-d319-4c01-83fe-0df57b41b69c" containerName="istio-proxy"
Nov 01 06:05:22.006582 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:22.006568    5689 cpu_manager.go:414] "containerMap: removing container" podUID="b9880a0a-c2b1-4a5e-a7b7-967038e13379" containerName="maas-api"
Nov 01 06:05:22.006582 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:22.006573    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9880a0a-c2b1-4a5e-a7b7-967038e13379" containerName="maas-api"
Nov 01 06:05:22.006582 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:22.006581    5689 cpu_manager.go:414] "containerMap: removing container" podUID="3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2" containerName="istio-proxy"
Nov 01 06:05:22.006582 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:22.006586    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2" containerName="istio-proxy"
Nov 01 06:05:22.006809 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:22.006604    5689 cpu_manager.go:414] "containerMap: removing container" podUID="f4ef14f8-4441-4dba-ace6-fad77c35a0b2" containerName="maas-api"
Nov 01 06:05:22.006809 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:22.006609    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4ef14f8-4441-4dba-ace6-fad77c35a0b2" containerName="maas-api"
Nov 01 06:05:22.006809 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:22.006620    5689 cpu_manager.go:414] "containerMap: removing container" podUID="9001d49f-9167-48e7-bacd-91219748d5e1" containerName="istio-proxy"
Nov 01 06:05:22.006809 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:22.006625    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9001d49f-9167-48e7-bacd-91219748d5e1" containerName="istio-proxy"
Nov 01 06:05:22.006809 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:22.006634    5689 cpu_manager.go:414] "containerMap: removing container" podUID="ea6d76c0-414c-4ae6-9c9a-01e41971592f" containerName="istio-proxy"
Nov 01 06:05:22.006809 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:22.006639    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea6d76c0-414c-4ae6-9c9a-01e41971592f" containerName="istio-proxy"
Nov 01 06:05:22.006809 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:22.006647    5689 cpu_manager.go:414] "containerMap: removing container" podUID="4a1fcaaf-0b04-4602-8985-866f6152303d" containerName="main"
Nov 01 06:05:22.006809 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:22.006652    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a1fcaaf-0b04-4602-8985-866f6152303d" containerName="main"
Nov 01 06:05:22.006809 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:22.006706    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea6d76c0-414c-4ae6-9c9a-01e41971592f" containerName="istio-proxy"
Nov 01 06:05:22.006809 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:22.006713    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7183c3f-d319-4c01-83fe-0df57b41b69c" containerName="istio-proxy"
Nov 01 06:05:22.006809 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:22.006719    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a1fcaaf-0b04-4602-8985-866f6152303d" containerName="main"
Nov 01 06:05:22.006809 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:22.006728    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9880a0a-c2b1-4a5e-a7b7-967038e13379" containerName="maas-api"
Nov 01 06:05:22.006809 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:22.006734    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fb62ef2-98a5-42ec-95ce-f18c1f3f81b2" containerName="istio-proxy"
Nov 01 06:05:22.006809 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:22.006742    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4ef14f8-4441-4dba-ace6-fad77c35a0b2" containerName="maas-api"
Nov 01 06:05:22.006809 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:22.006749    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="9001d49f-9167-48e7-bacd-91219748d5e1" containerName="istio-proxy"
Nov 01 06:05:22.009809 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:22.009794    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-9npk6"
Nov 01 06:05:22.103287 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:22.103249    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a008fc6-3439-4701-8748-c0d0928c0f8d-istio-data\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-9npk6\" (UID: \"9a008fc6-3439-4701-8748-c0d0928c0f8d\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-9npk6"
Nov 01 06:05:22.103527 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:22.103308    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a008fc6-3439-4701-8748-c0d0928c0f8d-workload-socket\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-9npk6\" (UID: \"9a008fc6-3439-4701-8748-c0d0928c0f8d\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-9npk6"
Nov 01 06:05:22.103527 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:22.103364    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a008fc6-3439-4701-8748-c0d0928c0f8d-credential-socket\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-9npk6\" (UID: \"9a008fc6-3439-4701-8748-c0d0928c0f8d\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-9npk6"
Nov 01 06:05:22.103527 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:22.103515    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a008fc6-3439-4701-8748-c0d0928c0f8d-istio-podinfo\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-9npk6\" (UID: \"9a008fc6-3439-4701-8748-c0d0928c0f8d\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-9npk6"
Nov 01 06:05:22.103727 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:22.103569    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a008fc6-3439-4701-8748-c0d0928c0f8d-istiod-ca-cert\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-9npk6\" (UID: \"9a008fc6-3439-4701-8748-c0d0928c0f8d\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-9npk6"
Nov 01 06:05:22.103727 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:22.103646    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a008fc6-3439-4701-8748-c0d0928c0f8d-istio-envoy\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-9npk6\" (UID: \"9a008fc6-3439-4701-8748-c0d0928c0f8d\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-9npk6"
Nov 01 06:05:22.103727 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:22.103677    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h74j4\" (UniqueName: \"kubernetes.io/projected/9a008fc6-3439-4701-8748-c0d0928c0f8d-kube-api-access-h74j4\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-9npk6\" (UID: \"9a008fc6-3439-4701-8748-c0d0928c0f8d\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-9npk6"
Nov 01 06:05:22.103727 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:22.103722    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a008fc6-3439-4701-8748-c0d0928c0f8d-istio-token\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-9npk6\" (UID: \"9a008fc6-3439-4701-8748-c0d0928c0f8d\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-9npk6"
Nov 01 06:05:22.103939 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:22.103751    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a008fc6-3439-4701-8748-c0d0928c0f8d-workload-certs\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-9npk6\" (UID: \"9a008fc6-3439-4701-8748-c0d0928c0f8d\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-9npk6"
Nov 01 06:05:22.131357 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:22.131323    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-9npk6"]
Nov 01 06:05:22.143112 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:22.143066    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:05:22.147569 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:22.147547    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-7d1ec3-openshift-default-dockercfg-thhxl\""
Nov 01 06:05:22.204947 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:22.204907    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a008fc6-3439-4701-8748-c0d0928c0f8d-istio-data\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-9npk6\" (UID: \"9a008fc6-3439-4701-8748-c0d0928c0f8d\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-9npk6"
Nov 01 06:05:22.205102 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:22.204966    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a008fc6-3439-4701-8748-c0d0928c0f8d-workload-socket\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-9npk6\" (UID: \"9a008fc6-3439-4701-8748-c0d0928c0f8d\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-9npk6"
Nov 01 06:05:22.205102 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:22.205022    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a008fc6-3439-4701-8748-c0d0928c0f8d-credential-socket\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-9npk6\" (UID: \"9a008fc6-3439-4701-8748-c0d0928c0f8d\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-9npk6"
Nov 01 06:05:22.205102 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:22.205048    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a008fc6-3439-4701-8748-c0d0928c0f8d-istio-podinfo\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-9npk6\" (UID: \"9a008fc6-3439-4701-8748-c0d0928c0f8d\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-9npk6"
Nov 01 06:05:22.205102 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:22.205074    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a008fc6-3439-4701-8748-c0d0928c0f8d-istiod-ca-cert\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-9npk6\" (UID: \"9a008fc6-3439-4701-8748-c0d0928c0f8d\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-9npk6"
Nov 01 06:05:22.205323 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:22.205126    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a008fc6-3439-4701-8748-c0d0928c0f8d-istio-envoy\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-9npk6\" (UID: \"9a008fc6-3439-4701-8748-c0d0928c0f8d\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-9npk6"
Nov 01 06:05:22.205323 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:22.205158    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h74j4\" (UniqueName: \"kubernetes.io/projected/9a008fc6-3439-4701-8748-c0d0928c0f8d-kube-api-access-h74j4\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-9npk6\" (UID: \"9a008fc6-3439-4701-8748-c0d0928c0f8d\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-9npk6"
Nov 01 06:05:22.205323 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:22.205199    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a008fc6-3439-4701-8748-c0d0928c0f8d-istio-token\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-9npk6\" (UID: \"9a008fc6-3439-4701-8748-c0d0928c0f8d\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-9npk6"
Nov 01 06:05:22.205323 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:22.205227    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a008fc6-3439-4701-8748-c0d0928c0f8d-workload-certs\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-9npk6\" (UID: \"9a008fc6-3439-4701-8748-c0d0928c0f8d\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-9npk6"
Nov 01 06:05:22.205741 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:22.205720    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a008fc6-3439-4701-8748-c0d0928c0f8d-workload-certs\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-9npk6\" (UID: \"9a008fc6-3439-4701-8748-c0d0928c0f8d\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-9npk6"
Nov 01 06:05:22.205937 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:22.205912    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a008fc6-3439-4701-8748-c0d0928c0f8d-istio-data\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-9npk6\" (UID: \"9a008fc6-3439-4701-8748-c0d0928c0f8d\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-9npk6"
Nov 01 06:05:22.206364 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:22.206337    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a008fc6-3439-4701-8748-c0d0928c0f8d-istiod-ca-cert\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-9npk6\" (UID: \"9a008fc6-3439-4701-8748-c0d0928c0f8d\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-9npk6"
Nov 01 06:05:22.206644 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:22.206619    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a008fc6-3439-4701-8748-c0d0928c0f8d-credential-socket\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-9npk6\" (UID: \"9a008fc6-3439-4701-8748-c0d0928c0f8d\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-9npk6"
Nov 01 06:05:22.207329 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:22.206869    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a008fc6-3439-4701-8748-c0d0928c0f8d-workload-socket\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-9npk6\" (UID: \"9a008fc6-3439-4701-8748-c0d0928c0f8d\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-9npk6"
Nov 01 06:05:22.209757 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:22.209737    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a008fc6-3439-4701-8748-c0d0928c0f8d-istio-envoy\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-9npk6\" (UID: \"9a008fc6-3439-4701-8748-c0d0928c0f8d\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-9npk6"
Nov 01 06:05:22.210253 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:22.210228    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a008fc6-3439-4701-8748-c0d0928c0f8d-istio-podinfo\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-9npk6\" (UID: \"9a008fc6-3439-4701-8748-c0d0928c0f8d\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-9npk6"
Nov 01 06:05:22.267024 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:22.266951    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a008fc6-3439-4701-8748-c0d0928c0f8d-istio-token\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-9npk6\" (UID: \"9a008fc6-3439-4701-8748-c0d0928c0f8d\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-9npk6"
Nov 01 06:05:22.267158 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:22.267140    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h74j4\" (UniqueName: \"kubernetes.io/projected/9a008fc6-3439-4701-8748-c0d0928c0f8d-kube-api-access-h74j4\") pod \"e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-9npk6\" (UID: \"9a008fc6-3439-4701-8748-c0d0928c0f8d\") " pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-9npk6"
Nov 01 06:05:22.323070 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:22.323039    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-9npk6"
Nov 01 06:05:22.325594 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:22.325570    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:05:22.752388 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:22.752357    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-9npk6"]
Nov 01 06:05:22.756146 ip-10-0-129-92 kubenswrapper[5689]: W1101 06:05:22.756117    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a008fc6_3439_4701_8748_c0d0928c0f8d.slice/crio-cd8a2f7c370f0abdff6e104aa67444f4767bfc8810b86c180ec20839047de029 WatchSource:0}: Error finding container cd8a2f7c370f0abdff6e104aa67444f4767bfc8810b86c180ec20839047de029: Status 404 returned error can't find the container with id cd8a2f7c370f0abdff6e104aa67444f4767bfc8810b86c180ec20839047de029
Nov 01 06:05:22.758191 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:22.758161    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 06:05:22.758269 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:22.758224    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 06:05:22.758269 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:22.758251    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 06:05:22.773387 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:22.773356    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-9npk6" event={"ID":"9a008fc6-3439-4701-8748-c0d0928c0f8d","Type":"ContainerStarted","Data":"cd8a2f7c370f0abdff6e104aa67444f4767bfc8810b86c180ec20839047de029"}
Nov 01 06:05:23.089101 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:23.088908    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:05:23.803204 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:23.803167    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-9npk6" event={"ID":"9a008fc6-3439-4701-8748-c0d0928c0f8d","Type":"ContainerStarted","Data":"c81c9cbe78137ac5eb906d05b01f5d33a2656f404dd9efcac2c45872eb94edfe"}
Nov 01 06:05:24.059734 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:24.059628    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-9npk6" podStartSLOduration=3.059614769 podStartE2EDuration="3.059614769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:05:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:05:23.997467097 +0000 UTC m=+3702.068263811" watchObservedRunningTime="2026-11-01 06:05:24.059614769 +0000 UTC m=+3702.130411488"
Nov 01 06:05:24.060661 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:24.060643    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:05:24.324038 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:24.323961    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-9npk6"
Nov 01 06:05:24.325691 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:24.325662    5689 patch_prober.go:29] interesting pod/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-9npk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body=
Nov 01 06:05:24.325844 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:24.325731    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-9npk6" podUID="9a008fc6-3439-4701-8748-c0d0928c0f8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused"
Nov 01 06:05:25.306864 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:25.306825    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:05:25.324373 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:25.324337    5689 patch_prober.go:29] interesting pod/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-9npk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body=
Nov 01 06:05:25.324843 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:25.324398    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-9npk6" podUID="9a008fc6-3439-4701-8748-c0d0928c0f8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused"
Nov 01 06:05:25.765877 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:25.765844    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-9npk6"]
Nov 01 06:05:25.808636 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:25.808601    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:05:25.813489 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:25.813441    5689 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-9npk6" secret="" err="secret \"e2e-worker-w0-7d1ec3-openshift-default-dockercfg-thhxl\" not found"
Nov 01 06:05:26.323379 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:26.323345    5689 patch_prober.go:29] interesting pod/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-9npk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body=
Nov 01 06:05:26.323578 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:26.323402    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-9npk6" podUID="9a008fc6-3439-4701-8748-c0d0928c0f8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused"
Nov 01 06:05:26.818707 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:26.818669    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-9npk6" podUID="9a008fc6-3439-4701-8748-c0d0928c0f8d" containerName="istio-proxy" containerID="cri-o://c81c9cbe78137ac5eb906d05b01f5d33a2656f404dd9efcac2c45872eb94edfe" gracePeriod=30
Nov 01 06:05:27.094027 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:27.093943    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:05:28.779533 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:28.779501    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:05:28.955480 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:28.955435    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:05:30.068672 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:05:30.068633    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 06:05:32.119196 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:32.119160    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:05:34.362304 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:34.362282    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-9npk6"
Nov 01 06:05:34.525277 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:34.525199    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a008fc6-3439-4701-8748-c0d0928c0f8d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a008fc6-3439-4701-8748-c0d0928c0f8d-workload-certs\") pod \"9a008fc6-3439-4701-8748-c0d0928c0f8d\" (UID: \"9a008fc6-3439-4701-8748-c0d0928c0f8d\") "
Nov 01 06:05:34.525526 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:34.525273    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a008fc6-3439-4701-8748-c0d0928c0f8d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a008fc6-3439-4701-8748-c0d0928c0f8d-credential-socket\") pod \"9a008fc6-3439-4701-8748-c0d0928c0f8d\" (UID: \"9a008fc6-3439-4701-8748-c0d0928c0f8d\") "
Nov 01 06:05:34.525526 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:34.525301    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a008fc6-3439-4701-8748-c0d0928c0f8d-istio-token\" (UniqueName: \"kubernetes.io/projected/9a008fc6-3439-4701-8748-c0d0928c0f8d-istio-token\") pod \"9a008fc6-3439-4701-8748-c0d0928c0f8d\" (UID: \"9a008fc6-3439-4701-8748-c0d0928c0f8d\") "
Nov 01 06:05:34.525526 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:34.525339    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a008fc6-3439-4701-8748-c0d0928c0f8d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a008fc6-3439-4701-8748-c0d0928c0f8d-istiod-ca-cert\") pod \"9a008fc6-3439-4701-8748-c0d0928c0f8d\" (UID: \"9a008fc6-3439-4701-8748-c0d0928c0f8d\") "
Nov 01 06:05:34.525526 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:34.525379    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a008fc6-3439-4701-8748-c0d0928c0f8d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a008fc6-3439-4701-8748-c0d0928c0f8d-istio-podinfo\") pod \"9a008fc6-3439-4701-8748-c0d0928c0f8d\" (UID: \"9a008fc6-3439-4701-8748-c0d0928c0f8d\") "
Nov 01 06:05:34.525526 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:34.525408    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a008fc6-3439-4701-8748-c0d0928c0f8d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a008fc6-3439-4701-8748-c0d0928c0f8d-istio-envoy\") pod \"9a008fc6-3439-4701-8748-c0d0928c0f8d\" (UID: \"9a008fc6-3439-4701-8748-c0d0928c0f8d\") "
Nov 01 06:05:34.525526 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:34.525480    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a008fc6-3439-4701-8748-c0d0928c0f8d-workload-certs" pod "9a008fc6-3439-4701-8748-c0d0928c0f8d" (UID: "9a008fc6-3439-4701-8748-c0d0928c0f8d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:05:34.525526 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:34.525510    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a008fc6-3439-4701-8748-c0d0928c0f8d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a008fc6-3439-4701-8748-c0d0928c0f8d-istio-data\") pod \"9a008fc6-3439-4701-8748-c0d0928c0f8d\" (UID: \"9a008fc6-3439-4701-8748-c0d0928c0f8d\") "
Nov 01 06:05:34.525912 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:34.525611    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a008fc6-3439-4701-8748-c0d0928c0f8d-credential-socket" pod "9a008fc6-3439-4701-8748-c0d0928c0f8d" (UID: "9a008fc6-3439-4701-8748-c0d0928c0f8d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:05:34.525912 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:34.525635    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a008fc6-3439-4701-8748-c0d0928c0f8d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a008fc6-3439-4701-8748-c0d0928c0f8d-workload-socket\") pod \"9a008fc6-3439-4701-8748-c0d0928c0f8d\" (UID: \"9a008fc6-3439-4701-8748-c0d0928c0f8d\") "
Nov 01 06:05:34.525912 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:34.525674    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a008fc6-3439-4701-8748-c0d0928c0f8d-kube-api-access-h74j4\" (UniqueName: \"kubernetes.io/projected/9a008fc6-3439-4701-8748-c0d0928c0f8d-kube-api-access-h74j4\") pod \"9a008fc6-3439-4701-8748-c0d0928c0f8d\" (UID: \"9a008fc6-3439-4701-8748-c0d0928c0f8d\") "
Nov 01 06:05:34.525912 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:34.525776    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a008fc6-3439-4701-8748-c0d0928c0f8d-istio-data" pod "9a008fc6-3439-4701-8748-c0d0928c0f8d" (UID: "9a008fc6-3439-4701-8748-c0d0928c0f8d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:05:34.525912 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:34.525828    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a008fc6-3439-4701-8748-c0d0928c0f8d-istiod-ca-cert" pod "9a008fc6-3439-4701-8748-c0d0928c0f8d" (UID: "9a008fc6-3439-4701-8748-c0d0928c0f8d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:05:34.525912 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:34.525847    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a008fc6-3439-4701-8748-c0d0928c0f8d-workload-socket" pod "9a008fc6-3439-4701-8748-c0d0928c0f8d" (UID: "9a008fc6-3439-4701-8748-c0d0928c0f8d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:05:34.526403 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:34.526070    5689 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a008fc6-3439-4701-8748-c0d0928c0f8d-istio-data\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:05:34.526403 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:34.526083    5689 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a008fc6-3439-4701-8748-c0d0928c0f8d-workload-socket\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:05:34.526403 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:34.526093    5689 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a008fc6-3439-4701-8748-c0d0928c0f8d-workload-certs\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:05:34.526403 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:34.526104    5689 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a008fc6-3439-4701-8748-c0d0928c0f8d-credential-socket\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:05:34.526403 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:34.526118    5689 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a008fc6-3439-4701-8748-c0d0928c0f8d-istiod-ca-cert\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:05:34.527791 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:34.527764    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a008fc6-3439-4701-8748-c0d0928c0f8d-istio-envoy" pod "9a008fc6-3439-4701-8748-c0d0928c0f8d" (UID: "9a008fc6-3439-4701-8748-c0d0928c0f8d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:05:34.527934 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:34.527916    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a008fc6-3439-4701-8748-c0d0928c0f8d-kube-api-access-h74j4" pod "9a008fc6-3439-4701-8748-c0d0928c0f8d" (UID: "9a008fc6-3439-4701-8748-c0d0928c0f8d"). InnerVolumeSpecName "kube-api-access-h74j4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:05:34.528083 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:34.528062    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a008fc6-3439-4701-8748-c0d0928c0f8d-istio-podinfo" pod "9a008fc6-3439-4701-8748-c0d0928c0f8d" (UID: "9a008fc6-3439-4701-8748-c0d0928c0f8d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:05:34.528192 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:34.528081    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a008fc6-3439-4701-8748-c0d0928c0f8d-istio-token" pod "9a008fc6-3439-4701-8748-c0d0928c0f8d" (UID: "9a008fc6-3439-4701-8748-c0d0928c0f8d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:05:34.627578 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:34.627548    5689 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a008fc6-3439-4701-8748-c0d0928c0f8d-istio-token\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:05:34.627578 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:34.627572    5689 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a008fc6-3439-4701-8748-c0d0928c0f8d-istio-podinfo\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:05:34.627578 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:34.627582    5689 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a008fc6-3439-4701-8748-c0d0928c0f8d-istio-envoy\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:05:34.627779 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:34.627591    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h74j4\" (UniqueName: \"kubernetes.io/projected/9a008fc6-3439-4701-8748-c0d0928c0f8d-kube-api-access-h74j4\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:05:34.858412 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:34.858332    5689 generic.go:356] "Generic (PLEG): container finished" podID="9a008fc6-3439-4701-8748-c0d0928c0f8d" containerID="c81c9cbe78137ac5eb906d05b01f5d33a2656f404dd9efcac2c45872eb94edfe" exitCode=0
Nov 01 06:05:34.858412 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:34.858380    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-9npk6"
Nov 01 06:05:34.858638 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:34.858413    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-9npk6" event={"ID":"9a008fc6-3439-4701-8748-c0d0928c0f8d","Type":"ContainerDied","Data":"c81c9cbe78137ac5eb906d05b01f5d33a2656f404dd9efcac2c45872eb94edfe"}
Nov 01 06:05:34.858638 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:34.858490    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-9npk6" event={"ID":"9a008fc6-3439-4701-8748-c0d0928c0f8d","Type":"ContainerDied","Data":"cd8a2f7c370f0abdff6e104aa67444f4767bfc8810b86c180ec20839047de029"}
Nov 01 06:05:34.858638 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:34.858518    5689 scope.go:122] "RemoveContainer" containerID="c81c9cbe78137ac5eb906d05b01f5d33a2656f404dd9efcac2c45872eb94edfe"
Nov 01 06:05:34.868061 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:34.868045    5689 scope.go:122] "RemoveContainer" containerID="c81c9cbe78137ac5eb906d05b01f5d33a2656f404dd9efcac2c45872eb94edfe"
Nov 01 06:05:34.868341 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:05:34.868324    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c81c9cbe78137ac5eb906d05b01f5d33a2656f404dd9efcac2c45872eb94edfe\": container with ID starting with c81c9cbe78137ac5eb906d05b01f5d33a2656f404dd9efcac2c45872eb94edfe not found: ID does not exist" containerID="c81c9cbe78137ac5eb906d05b01f5d33a2656f404dd9efcac2c45872eb94edfe"
Nov 01 06:05:34.868395 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:34.868348    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c81c9cbe78137ac5eb906d05b01f5d33a2656f404dd9efcac2c45872eb94edfe"} err="failed to get container status \"c81c9cbe78137ac5eb906d05b01f5d33a2656f404dd9efcac2c45872eb94edfe\": rpc error: code = NotFound desc = could not find container \"c81c9cbe78137ac5eb906d05b01f5d33a2656f404dd9efcac2c45872eb94edfe\": container with ID starting with c81c9cbe78137ac5eb906d05b01f5d33a2656f404dd9efcac2c45872eb94edfe not found: ID does not exist"
Nov 01 06:05:35.023520 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:35.023494    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-9npk6"]
Nov 01 06:05:35.072873 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:35.072846    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7d1ec3-openshift-default-86c7d47c6c-9npk6"]
Nov 01 06:05:36.066496 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:36.066463    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a008fc6-3439-4701-8748-c0d0928c0f8d" path="/var/lib/kubelet/pods/9a008fc6-3439-4701-8748-c0d0928c0f8d/volumes"
Nov 01 06:05:37.305947 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:37.305910    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:05:40.584269 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:40.584236    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:05:41.624036 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:41.624004    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:05:41.801301 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:41.801266    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:05:42.034435 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:42.034410    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:05:46.796942 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:46.796906    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:05:51.502846 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:51.502791    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:05:52.700397 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:52.700363    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:05:56.315989 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:05:56.315955    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:06:00.068840 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:06:00.068804    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 06:06:01.113102 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:06:01.113069    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:06:05.810865 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:06:05.810836    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:06:10.538520 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:06:10.538479    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:06:15.102101 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:06:15.102070    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:06:15.412589 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:06:15.412559    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:06:20.204928 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:06:20.204898    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:06:25.017910 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:06:25.017883    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:06:29.716893 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:06:29.716860    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:06:30.068495 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:06:30.068392    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 06:06:34.404767 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:06:34.404738    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:06:34.578552 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:06:34.578522    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:06:39.112742 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:06:39.112711    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:06:43.894934 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:06:43.894894    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:06:48.696928 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:06:48.696893    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:06:53.504907 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:06:53.504836    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:06:58.198559 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:06:58.198524    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:07:00.068577 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:07:00.068545    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 06:07:00.196142 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:07:00.196114    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:07:02.909368 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:07:02.909337    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:07:07.720513 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:07:07.720475    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:07:12.533672 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:07:12.533633    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:07:12.575685 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:07:12.575653    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:07:15.442689 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:07:15.442654    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:07:17.228282 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:07:17.228248    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:07:22.000582 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:07:22.000552    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:07:25.166163 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:07:25.166131    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:07:25.611331 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:07:25.611250    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:07:26.800160 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:07:26.800130    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:07:30.067767 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:07:30.067735    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 06:07:31.318062 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:07:31.318031    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-897c87/sub-test-model-897c87-kserve-6d49ff64df-f27f4"]
Nov 01 06:07:31.365377 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:07:31.365344    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:07:31.544496 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:07:31.544427    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:07:36.408902 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:07:36.408864    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:07:40.884595 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:07:40.884561    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:07:41.114534 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:07:41.114498    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:07:45.836153 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:07:45.836120    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:07:46.265680 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:07:46.265638    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:07:46.338132 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:07:46.338078    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-897c87/sub-test-model-897c87-kserve-6d49ff64df-f27f4" podUID="451c733b-f9b7-4bc0-a113-a4e7607bba9f" containerName="main" containerID="cri-o://a0a4cd26fb3fe18f2c2f6cd114f6ce5f5dbf5c4429bf58ed367fbebbbbd84301" gracePeriod=45
Nov 01 06:07:46.446082 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:07:46.446049    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:07:46.477314 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:07:46.477287    5689 generic.go:356] "Generic (PLEG): container finished" podID="451c733b-f9b7-4bc0-a113-a4e7607bba9f" containerID="a0a4cd26fb3fe18f2c2f6cd114f6ce5f5dbf5c4429bf58ed367fbebbbbd84301" exitCode=0
Nov 01 06:07:46.477472 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:07:46.477360    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-897c87/sub-test-model-897c87-kserve-6d49ff64df-f27f4" event={"ID":"451c733b-f9b7-4bc0-a113-a4e7607bba9f","Type":"ContainerDied","Data":"a0a4cd26fb3fe18f2c2f6cd114f6ce5f5dbf5c4429bf58ed367fbebbbbd84301"}
Nov 01 06:07:46.599260 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:07:46.599238    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-897c87/sub-test-model-897c87-kserve-6d49ff64df-f27f4"
Nov 01 06:07:46.647544 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:07:46.647521    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/451c733b-f9b7-4bc0-a113-a4e7607bba9f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/451c733b-f9b7-4bc0-a113-a4e7607bba9f-model-cache\") pod \"451c733b-f9b7-4bc0-a113-a4e7607bba9f\" (UID: \"451c733b-f9b7-4bc0-a113-a4e7607bba9f\") "
Nov 01 06:07:46.647659 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:07:46.647574    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/451c733b-f9b7-4bc0-a113-a4e7607bba9f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/451c733b-f9b7-4bc0-a113-a4e7607bba9f-dshm\") pod \"451c733b-f9b7-4bc0-a113-a4e7607bba9f\" (UID: \"451c733b-f9b7-4bc0-a113-a4e7607bba9f\") "
Nov 01 06:07:46.647659 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:07:46.647596    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/451c733b-f9b7-4bc0-a113-a4e7607bba9f-home\" (UniqueName: \"kubernetes.io/empty-dir/451c733b-f9b7-4bc0-a113-a4e7607bba9f-home\") pod \"451c733b-f9b7-4bc0-a113-a4e7607bba9f\" (UID: \"451c733b-f9b7-4bc0-a113-a4e7607bba9f\") "
Nov 01 06:07:46.647659 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:07:46.647622    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/451c733b-f9b7-4bc0-a113-a4e7607bba9f-tls-certs\" (UniqueName: \"kubernetes.io/secret/451c733b-f9b7-4bc0-a113-a4e7607bba9f-tls-certs\") pod \"451c733b-f9b7-4bc0-a113-a4e7607bba9f\" (UID: \"451c733b-f9b7-4bc0-a113-a4e7607bba9f\") "
Nov 01 06:07:46.647659 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:07:46.647641    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/451c733b-f9b7-4bc0-a113-a4e7607bba9f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/451c733b-f9b7-4bc0-a113-a4e7607bba9f-tmp-dir\") pod \"451c733b-f9b7-4bc0-a113-a4e7607bba9f\" (UID: \"451c733b-f9b7-4bc0-a113-a4e7607bba9f\") "
Nov 01 06:07:46.647868 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:07:46.647679    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/451c733b-f9b7-4bc0-a113-a4e7607bba9f-kube-api-access-sb7pn\" (UniqueName: \"kubernetes.io/projected/451c733b-f9b7-4bc0-a113-a4e7607bba9f-kube-api-access-sb7pn\") pod \"451c733b-f9b7-4bc0-a113-a4e7607bba9f\" (UID: \"451c733b-f9b7-4bc0-a113-a4e7607bba9f\") "
Nov 01 06:07:46.647868 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:07:46.647802    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/451c733b-f9b7-4bc0-a113-a4e7607bba9f-model-cache" pod "451c733b-f9b7-4bc0-a113-a4e7607bba9f" (UID: "451c733b-f9b7-4bc0-a113-a4e7607bba9f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:07:46.647988 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:07:46.647903    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/451c733b-f9b7-4bc0-a113-a4e7607bba9f-home" pod "451c733b-f9b7-4bc0-a113-a4e7607bba9f" (UID: "451c733b-f9b7-4bc0-a113-a4e7607bba9f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:07:46.648048 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:07:46.647980    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/451c733b-f9b7-4bc0-a113-a4e7607bba9f-tmp-dir" pod "451c733b-f9b7-4bc0-a113-a4e7607bba9f" (UID: "451c733b-f9b7-4bc0-a113-a4e7607bba9f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:07:46.648126 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:07:46.648108    5689 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/451c733b-f9b7-4bc0-a113-a4e7607bba9f-model-cache\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:07:46.648180 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:07:46.648132    5689 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/451c733b-f9b7-4bc0-a113-a4e7607bba9f-home\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:07:46.648180 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:07:46.648173    5689 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/451c733b-f9b7-4bc0-a113-a4e7607bba9f-tmp-dir\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:07:46.649838 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:07:46.649812    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/451c733b-f9b7-4bc0-a113-a4e7607bba9f-tls-certs" pod "451c733b-f9b7-4bc0-a113-a4e7607bba9f" (UID: "451c733b-f9b7-4bc0-a113-a4e7607bba9f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:07:46.649946 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:07:46.649813    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/451c733b-f9b7-4bc0-a113-a4e7607bba9f-dshm" pod "451c733b-f9b7-4bc0-a113-a4e7607bba9f" (UID: "451c733b-f9b7-4bc0-a113-a4e7607bba9f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:07:46.649946 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:07:46.649852    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/451c733b-f9b7-4bc0-a113-a4e7607bba9f-kube-api-access-sb7pn" pod "451c733b-f9b7-4bc0-a113-a4e7607bba9f" (UID: "451c733b-f9b7-4bc0-a113-a4e7607bba9f"). InnerVolumeSpecName "kube-api-access-sb7pn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:07:46.748903 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:07:46.748876    5689 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/451c733b-f9b7-4bc0-a113-a4e7607bba9f-dshm\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:07:46.748903 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:07:46.748900    5689 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/451c733b-f9b7-4bc0-a113-a4e7607bba9f-tls-certs\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:07:46.749087 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:07:46.748911    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sb7pn\" (UniqueName: \"kubernetes.io/projected/451c733b-f9b7-4bc0-a113-a4e7607bba9f-kube-api-access-sb7pn\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:07:47.483283 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:07:47.483251    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-897c87/sub-test-model-897c87-kserve-6d49ff64df-f27f4"
Nov 01 06:07:47.483760 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:07:47.483283    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-897c87/sub-test-model-897c87-kserve-6d49ff64df-f27f4" event={"ID":"451c733b-f9b7-4bc0-a113-a4e7607bba9f","Type":"ContainerDied","Data":"b0d596388662c6d1b1e2049ad0daea9fee8859cb37daddaf1f1c3a5af030fb9c"}
Nov 01 06:07:47.483760 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:07:47.483332    5689 scope.go:122] "RemoveContainer" containerID="a0a4cd26fb3fe18f2c2f6cd114f6ce5f5dbf5c4429bf58ed367fbebbbbd84301"
Nov 01 06:07:47.697071 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:07:47.697041    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-897c87/sub-test-model-897c87-kserve-6d49ff64df-f27f4"]
Nov 01 06:07:47.700134 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:07:47.700100    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-897c87/sub-test-model-897c87-kserve-6d49ff64df-f27f4"]
Nov 01 06:07:48.066598 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:07:48.066567    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="451c733b-f9b7-4bc0-a113-a4e7607bba9f" path="/var/lib/kubelet/pods/451c733b-f9b7-4bc0-a113-a4e7607bba9f/volumes"
Nov 01 06:07:49.763947 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:07:49.763896    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:07:50.278604 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:07:50.278569    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:07:50.710516 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:07:50.710479    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:07:55.408591 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:07:55.408558    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:07:55.975823 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:07:55.975788    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:07:56.078405 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:07:56.078379    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:07:56.164698 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:07:56.164669    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:07:56.776565 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:07:56.776534    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:07:59.727298 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:07:59.727267    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-897c87-588d6567c7-7nvdj"]
Nov 01 06:07:59.727810 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:07:59.727443    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-897c87-588d6567c7-7nvdj" podUID="3d1f1368-d8ef-480f-916c-27fb53b44b25" containerName="maas-api" containerID="cri-o://780937dc013a34c53e9a9a0091368dfd84c88158a416240d078b7ee2a661589a" gracePeriod=30
Nov 01 06:07:59.859305 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:07:59.859260    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:07:59.981045 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:07:59.980989    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-897c87-588d6567c7-7nvdj"
Nov 01 06:08:00.067624 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:00.067598    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d1f1368-d8ef-480f-916c-27fb53b44b25-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d1f1368-d8ef-480f-916c-27fb53b44b25-openshift-service-ca\") pod \"3d1f1368-d8ef-480f-916c-27fb53b44b25\" (UID: \"3d1f1368-d8ef-480f-916c-27fb53b44b25\") "
Nov 01 06:08:00.067797 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:00.067653    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d1f1368-d8ef-480f-916c-27fb53b44b25-kube-api-access-ls5pp\" (UniqueName: \"kubernetes.io/projected/3d1f1368-d8ef-480f-916c-27fb53b44b25-kube-api-access-ls5pp\") pod \"3d1f1368-d8ef-480f-916c-27fb53b44b25\" (UID: \"3d1f1368-d8ef-480f-916c-27fb53b44b25\") "
Nov 01 06:08:00.067797 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:00.067723    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d1f1368-d8ef-480f-916c-27fb53b44b25-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d1f1368-d8ef-480f-916c-27fb53b44b25-metrics-certs\") pod \"3d1f1368-d8ef-480f-916c-27fb53b44b25\" (UID: \"3d1f1368-d8ef-480f-916c-27fb53b44b25\") "
Nov 01 06:08:00.067895 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:00.067828    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d1f1368-d8ef-480f-916c-27fb53b44b25-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d1f1368-d8ef-480f-916c-27fb53b44b25-maas-api-tls\") pod \"3d1f1368-d8ef-480f-916c-27fb53b44b25\" (UID: \"3d1f1368-d8ef-480f-916c-27fb53b44b25\") "
Nov 01 06:08:00.067990 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:00.067966    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d1f1368-d8ef-480f-916c-27fb53b44b25-openshift-service-ca" pod "3d1f1368-d8ef-480f-916c-27fb53b44b25" (UID: "3d1f1368-d8ef-480f-916c-27fb53b44b25"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:08:00.068156 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:00.068138    5689 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d1f1368-d8ef-480f-916c-27fb53b44b25-openshift-service-ca\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:08:00.068962 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:08:00.068875    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 06:08:00.069871 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:00.069849    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d1f1368-d8ef-480f-916c-27fb53b44b25-kube-api-access-ls5pp" pod "3d1f1368-d8ef-480f-916c-27fb53b44b25" (UID: "3d1f1368-d8ef-480f-916c-27fb53b44b25"). InnerVolumeSpecName "kube-api-access-ls5pp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:08:00.070046 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:00.070033    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d1f1368-d8ef-480f-916c-27fb53b44b25-metrics-certs" pod "3d1f1368-d8ef-480f-916c-27fb53b44b25" (UID: "3d1f1368-d8ef-480f-916c-27fb53b44b25"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:08:00.070403 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:00.070385    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d1f1368-d8ef-480f-916c-27fb53b44b25-maas-api-tls" pod "3d1f1368-d8ef-480f-916c-27fb53b44b25" (UID: "3d1f1368-d8ef-480f-916c-27fb53b44b25"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:08:00.169692 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:00.169662    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ls5pp\" (UniqueName: \"kubernetes.io/projected/3d1f1368-d8ef-480f-916c-27fb53b44b25-kube-api-access-ls5pp\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:08:00.169692 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:00.169686    5689 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d1f1368-d8ef-480f-916c-27fb53b44b25-metrics-certs\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:08:00.169921 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:00.169700    5689 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d1f1368-d8ef-480f-916c-27fb53b44b25-maas-api-tls\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:08:00.543515 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:00.543476    5689 generic.go:356] "Generic (PLEG): container finished" podID="3d1f1368-d8ef-480f-916c-27fb53b44b25" containerID="780937dc013a34c53e9a9a0091368dfd84c88158a416240d078b7ee2a661589a" exitCode=0
Nov 01 06:08:00.543692 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:00.543528    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-897c87-588d6567c7-7nvdj"
Nov 01 06:08:00.543692 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:00.543561    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-897c87-588d6567c7-7nvdj" event={"ID":"3d1f1368-d8ef-480f-916c-27fb53b44b25","Type":"ContainerDied","Data":"780937dc013a34c53e9a9a0091368dfd84c88158a416240d078b7ee2a661589a"}
Nov 01 06:08:00.543692 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:00.543637    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-897c87-588d6567c7-7nvdj" event={"ID":"3d1f1368-d8ef-480f-916c-27fb53b44b25","Type":"ContainerDied","Data":"5e937fcc8cf650201de6ba1a236585fabd8b726609cd85f32015e801ae9ee057"}
Nov 01 06:08:00.543692 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:00.543658    5689 scope.go:122] "RemoveContainer" containerID="780937dc013a34c53e9a9a0091368dfd84c88158a416240d078b7ee2a661589a"
Nov 01 06:08:00.552580 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:00.552507    5689 scope.go:122] "RemoveContainer" containerID="780937dc013a34c53e9a9a0091368dfd84c88158a416240d078b7ee2a661589a"
Nov 01 06:08:00.552797 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:08:00.552776    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"780937dc013a34c53e9a9a0091368dfd84c88158a416240d078b7ee2a661589a\": container with ID starting with 780937dc013a34c53e9a9a0091368dfd84c88158a416240d078b7ee2a661589a not found: ID does not exist" containerID="780937dc013a34c53e9a9a0091368dfd84c88158a416240d078b7ee2a661589a"
Nov 01 06:08:00.552882 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:00.552802    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"780937dc013a34c53e9a9a0091368dfd84c88158a416240d078b7ee2a661589a"} err="failed to get container status \"780937dc013a34c53e9a9a0091368dfd84c88158a416240d078b7ee2a661589a\": rpc error: code = NotFound desc = could not find container \"780937dc013a34c53e9a9a0091368dfd84c88158a416240d078b7ee2a661589a\": container with ID starting with 780937dc013a34c53e9a9a0091368dfd84c88158a416240d078b7ee2a661589a not found: ID does not exist"
Nov 01 06:08:00.705885 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:00.705855    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-897c87-588d6567c7-7nvdj"]
Nov 01 06:08:00.754933 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:00.754902    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-897c87-588d6567c7-7nvdj"]
Nov 01 06:08:01.692726 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:01.692693    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:08:02.066273 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:02.066194    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d1f1368-d8ef-480f-916c-27fb53b44b25" path="/var/lib/kubelet/pods/3d1f1368-d8ef-480f-916c-27fb53b44b25/volumes"
Nov 01 06:08:02.645022 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:02.644994    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:08:02.815371 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:02.815343    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:08:04.213664 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:04.213631    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:08:04.914196 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:04.914167    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:08:05.792436 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:05.792402    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:08:06.734854 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:06.734820    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:08:11.187198 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:11.187166    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:08:11.584206 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:11.584130    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:08:11.756915 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:11.756885    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:08:12.297095 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:12.297056    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-fd2672-659f849fc9-pqzgj"]
Nov 01 06:08:12.402525 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:12.402492    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-fd2672-7d58df9c59-2glzq"]
Nov 01 06:08:12.704329 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:12.704299    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:08:12.882084 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:12.882050    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:08:15.783619 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:15.783590    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:08:16.698321 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:16.698283    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:08:17.297566 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:17.297514    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-fd2672-659f849fc9-pqzgj" podUID="75a0b394-e687-407f-8a86-4f2100a6dfc6" containerName="payload-processing" containerID="cri-o://5cadde8ea56c19d44f37d4191e0e4be3b02ad73dc04cca43cdddffe095d008d4" gracePeriod=25
Nov 01 06:08:17.353647 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:17.353607    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:08:17.403484 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:17.403400    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-fd2672-7d58df9c59-2glzq" podUID="0dc7a0fa-fa53-4c98-a342-b3f67b3a1824" containerName="payload-pre-processing" containerID="cri-o://f8406bc94f62b89ad46b1c26b1ce28a978b1881a7383fd2523ed439390165a06" gracePeriod=25
Nov 01 06:08:20.745169 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:20.743973    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fd2672-fd4cfd8c5-zs46l"]
Nov 01 06:08:20.745169 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:20.744320    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fd2672-fd4cfd8c5-zs46l" podUID="f9e983e7-105f-4706-8872-3b3893f748f0" containerName="maas-api" containerID="cri-o://dd6c68f4fc24183ad07b37f7394e968330d34a6a68c8117a4fcbe0f8925934e6" gracePeriod=30
Nov 01 06:08:20.809065 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:08:20.809028    5689 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9e983e7_105f_4706_8872_3b3893f748f0.slice/crio-conmon-dd6c68f4fc24183ad07b37f7394e968330d34a6a68c8117a4fcbe0f8925934e6.scope\": RecentStats: unable to find data in memory cache]"
Nov 01 06:08:20.879014 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:20.878980    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:08:21.014091 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:21.014069    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fd2672-fd4cfd8c5-zs46l"
Nov 01 06:08:21.170817 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:21.170791    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9e983e7-105f-4706-8872-3b3893f748f0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9e983e7-105f-4706-8872-3b3893f748f0-openshift-service-ca\") pod \"f9e983e7-105f-4706-8872-3b3893f748f0\" (UID: \"f9e983e7-105f-4706-8872-3b3893f748f0\") "
Nov 01 06:08:21.171016 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:21.170855    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9e983e7-105f-4706-8872-3b3893f748f0-kube-api-access-l9w8p\" (UniqueName: \"kubernetes.io/projected/f9e983e7-105f-4706-8872-3b3893f748f0-kube-api-access-l9w8p\") pod \"f9e983e7-105f-4706-8872-3b3893f748f0\" (UID: \"f9e983e7-105f-4706-8872-3b3893f748f0\") "
Nov 01 06:08:21.171016 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:21.170881    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9e983e7-105f-4706-8872-3b3893f748f0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9e983e7-105f-4706-8872-3b3893f748f0-metrics-certs\") pod \"f9e983e7-105f-4706-8872-3b3893f748f0\" (UID: \"f9e983e7-105f-4706-8872-3b3893f748f0\") "
Nov 01 06:08:21.171016 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:21.170955    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9e983e7-105f-4706-8872-3b3893f748f0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9e983e7-105f-4706-8872-3b3893f748f0-maas-api-tls\") pod \"f9e983e7-105f-4706-8872-3b3893f748f0\" (UID: \"f9e983e7-105f-4706-8872-3b3893f748f0\") "
Nov 01 06:08:21.171255 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:21.171225    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9e983e7-105f-4706-8872-3b3893f748f0-openshift-service-ca" pod "f9e983e7-105f-4706-8872-3b3893f748f0" (UID: "f9e983e7-105f-4706-8872-3b3893f748f0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:08:21.171829 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:21.171565    5689 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9e983e7-105f-4706-8872-3b3893f748f0-openshift-service-ca\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:08:21.173164 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:21.173144    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9e983e7-105f-4706-8872-3b3893f748f0-kube-api-access-l9w8p" pod "f9e983e7-105f-4706-8872-3b3893f748f0" (UID: "f9e983e7-105f-4706-8872-3b3893f748f0"). InnerVolumeSpecName "kube-api-access-l9w8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:08:21.173723 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:21.173697    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9e983e7-105f-4706-8872-3b3893f748f0-metrics-certs" pod "f9e983e7-105f-4706-8872-3b3893f748f0" (UID: "f9e983e7-105f-4706-8872-3b3893f748f0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:08:21.173723 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:21.173712    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9e983e7-105f-4706-8872-3b3893f748f0-maas-api-tls" pod "f9e983e7-105f-4706-8872-3b3893f748f0" (UID: "f9e983e7-105f-4706-8872-3b3893f748f0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:08:21.272547 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:21.272433    5689 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9e983e7-105f-4706-8872-3b3893f748f0-maas-api-tls\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:08:21.272547 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:21.272495    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9w8p\" (UniqueName: \"kubernetes.io/projected/f9e983e7-105f-4706-8872-3b3893f748f0-kube-api-access-l9w8p\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:08:21.272547 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:21.272510    5689 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9e983e7-105f-4706-8872-3b3893f748f0-metrics-certs\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:08:21.641478 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:21.641367    5689 generic.go:356] "Generic (PLEG): container finished" podID="f9e983e7-105f-4706-8872-3b3893f748f0" containerID="dd6c68f4fc24183ad07b37f7394e968330d34a6a68c8117a4fcbe0f8925934e6" exitCode=0
Nov 01 06:08:21.641478 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:21.641415    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fd2672-fd4cfd8c5-zs46l"
Nov 01 06:08:21.641716 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:21.641471    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fd2672-fd4cfd8c5-zs46l" event={"ID":"f9e983e7-105f-4706-8872-3b3893f748f0","Type":"ContainerDied","Data":"dd6c68f4fc24183ad07b37f7394e968330d34a6a68c8117a4fcbe0f8925934e6"}
Nov 01 06:08:21.641716 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:21.641526    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fd2672-fd4cfd8c5-zs46l" event={"ID":"f9e983e7-105f-4706-8872-3b3893f748f0","Type":"ContainerDied","Data":"35812afc38b8574c33bebfcd3d097e43b53b66e74b036c93942b79488c601373"}
Nov 01 06:08:21.641716 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:21.641547    5689 scope.go:122] "RemoveContainer" containerID="dd6c68f4fc24183ad07b37f7394e968330d34a6a68c8117a4fcbe0f8925934e6"
Nov 01 06:08:21.652623 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:21.651856    5689 scope.go:122] "RemoveContainer" containerID="dd6c68f4fc24183ad07b37f7394e968330d34a6a68c8117a4fcbe0f8925934e6"
Nov 01 06:08:21.652623 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:08:21.652204    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd6c68f4fc24183ad07b37f7394e968330d34a6a68c8117a4fcbe0f8925934e6\": container with ID starting with dd6c68f4fc24183ad07b37f7394e968330d34a6a68c8117a4fcbe0f8925934e6 not found: ID does not exist" containerID="dd6c68f4fc24183ad07b37f7394e968330d34a6a68c8117a4fcbe0f8925934e6"
Nov 01 06:08:21.652623 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:21.652244    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd6c68f4fc24183ad07b37f7394e968330d34a6a68c8117a4fcbe0f8925934e6"} err="failed to get container status \"dd6c68f4fc24183ad07b37f7394e968330d34a6a68c8117a4fcbe0f8925934e6\": rpc error: code = NotFound desc = could not find container \"dd6c68f4fc24183ad07b37f7394e968330d34a6a68c8117a4fcbe0f8925934e6\": container with ID starting with dd6c68f4fc24183ad07b37f7394e968330d34a6a68c8117a4fcbe0f8925934e6 not found: ID does not exist"
Nov 01 06:08:21.804190 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:21.804160    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fd2672-fd4cfd8c5-zs46l"]
Nov 01 06:08:21.853705 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:21.853677    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fd2672-fd4cfd8c5-zs46l"]
Nov 01 06:08:22.065825 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:22.065794    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9e983e7-105f-4706-8872-3b3893f748f0" path="/var/lib/kubelet/pods/f9e983e7-105f-4706-8872-3b3893f748f0/volumes"
Nov 01 06:08:22.607768 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:22.607735    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:08:23.238987 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:23.238940    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:08:23.419539 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:23.419502    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:08:24.814618 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:24.814584    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:08:25.364745 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:25.364714    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:08:25.513177 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:25.513140    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-fd2672-659f849fc9-pqzgj" podUID="75a0b394-e687-407f-8a86-4f2100a6dfc6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:08:25.513177 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:25.513161    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-fd2672-7d58df9c59-2glzq" podUID="0dc7a0fa-fa53-4c98-a342-b3f67b3a1824" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:08:26.276745 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:26.276710    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:08:27.396354 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:27.396319    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:08:27.697949 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:27.697915    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:08:30.067726 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:08:30.067690    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 06:08:32.195650 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:32.195619    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:08:32.366943 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:32.366909    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:08:32.553144 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:32.553065    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:08:32.719256 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:32.719228    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:08:35.513327 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:35.513289    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-fd2672-7d58df9c59-2glzq" podUID="0dc7a0fa-fa53-4c98-a342-b3f67b3a1824" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:08:35.513803 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:35.513354    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-fd2672-659f849fc9-pqzgj" podUID="75a0b394-e687-407f-8a86-4f2100a6dfc6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:08:37.177191 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:37.177160    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:08:37.571378 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:37.571348    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-fd2672-659f849fc9-pqzgj"
Nov 01 06:08:37.612181 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:37.612153    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75a0b394-e687-407f-8a86-4f2100a6dfc6-kube-api-access-rw4pp\" (UniqueName: \"kubernetes.io/projected/75a0b394-e687-407f-8a86-4f2100a6dfc6-kube-api-access-rw4pp\") pod \"75a0b394-e687-407f-8a86-4f2100a6dfc6\" (UID: \"75a0b394-e687-407f-8a86-4f2100a6dfc6\") "
Nov 01 06:08:37.612355 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:37.612192    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75a0b394-e687-407f-8a86-4f2100a6dfc6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75a0b394-e687-407f-8a86-4f2100a6dfc6-plugins-config-volume\") pod \"75a0b394-e687-407f-8a86-4f2100a6dfc6\" (UID: \"75a0b394-e687-407f-8a86-4f2100a6dfc6\") "
Nov 01 06:08:37.612647 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:37.612612    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75a0b394-e687-407f-8a86-4f2100a6dfc6-plugins-config-volume" pod "75a0b394-e687-407f-8a86-4f2100a6dfc6" (UID: "75a0b394-e687-407f-8a86-4f2100a6dfc6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:08:37.614520 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:37.614490    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75a0b394-e687-407f-8a86-4f2100a6dfc6-kube-api-access-rw4pp" pod "75a0b394-e687-407f-8a86-4f2100a6dfc6" (UID: "75a0b394-e687-407f-8a86-4f2100a6dfc6"). InnerVolumeSpecName "kube-api-access-rw4pp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:08:37.654591 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:37.654570    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-fd2672-7d58df9c59-2glzq"
Nov 01 06:08:37.712879 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:37.712848    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dc7a0fa-fa53-4c98-a342-b3f67b3a1824-kube-api-access-vjqmd\" (UniqueName: \"kubernetes.io/projected/0dc7a0fa-fa53-4c98-a342-b3f67b3a1824-kube-api-access-vjqmd\") pod \"0dc7a0fa-fa53-4c98-a342-b3f67b3a1824\" (UID: \"0dc7a0fa-fa53-4c98-a342-b3f67b3a1824\") "
Nov 01 06:08:37.713011 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:37.712935    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dc7a0fa-fa53-4c98-a342-b3f67b3a1824-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dc7a0fa-fa53-4c98-a342-b3f67b3a1824-plugins-config-volume\") pod \"0dc7a0fa-fa53-4c98-a342-b3f67b3a1824\" (UID: \"0dc7a0fa-fa53-4c98-a342-b3f67b3a1824\") "
Nov 01 06:08:37.713248 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:37.713230    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rw4pp\" (UniqueName: \"kubernetes.io/projected/75a0b394-e687-407f-8a86-4f2100a6dfc6-kube-api-access-rw4pp\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:08:37.713299 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:37.713255    5689 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75a0b394-e687-407f-8a86-4f2100a6dfc6-plugins-config-volume\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:08:37.713299 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:37.713283    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dc7a0fa-fa53-4c98-a342-b3f67b3a1824-plugins-config-volume" pod "0dc7a0fa-fa53-4c98-a342-b3f67b3a1824" (UID: "0dc7a0fa-fa53-4c98-a342-b3f67b3a1824"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:08:37.715176 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:37.715144    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dc7a0fa-fa53-4c98-a342-b3f67b3a1824-kube-api-access-vjqmd" pod "0dc7a0fa-fa53-4c98-a342-b3f67b3a1824" (UID: "0dc7a0fa-fa53-4c98-a342-b3f67b3a1824"). InnerVolumeSpecName "kube-api-access-vjqmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:08:37.717752 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:37.717729    5689 generic.go:356] "Generic (PLEG): container finished" podID="75a0b394-e687-407f-8a86-4f2100a6dfc6" containerID="5cadde8ea56c19d44f37d4191e0e4be3b02ad73dc04cca43cdddffe095d008d4" exitCode=0
Nov 01 06:08:37.717865 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:37.717779    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-fd2672-659f849fc9-pqzgj"
Nov 01 06:08:37.717865 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:37.717809    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-fd2672-659f849fc9-pqzgj" event={"ID":"75a0b394-e687-407f-8a86-4f2100a6dfc6","Type":"ContainerDied","Data":"5cadde8ea56c19d44f37d4191e0e4be3b02ad73dc04cca43cdddffe095d008d4"}
Nov 01 06:08:37.717865 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:37.717856    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-fd2672-659f849fc9-pqzgj" event={"ID":"75a0b394-e687-407f-8a86-4f2100a6dfc6","Type":"ContainerDied","Data":"ff7e6339ff7529e872858962d2a931c19b053a752f4e9c6d267496b2ee14173c"}
Nov 01 06:08:37.718044 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:37.717877    5689 scope.go:122] "RemoveContainer" containerID="5cadde8ea56c19d44f37d4191e0e4be3b02ad73dc04cca43cdddffe095d008d4"
Nov 01 06:08:37.719229 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:37.719205    5689 generic.go:356] "Generic (PLEG): container finished" podID="0dc7a0fa-fa53-4c98-a342-b3f67b3a1824" containerID="f8406bc94f62b89ad46b1c26b1ce28a978b1881a7383fd2523ed439390165a06" exitCode=0
Nov 01 06:08:37.719331 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:37.719255    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-fd2672-7d58df9c59-2glzq"
Nov 01 06:08:37.719331 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:37.719292    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-fd2672-7d58df9c59-2glzq" event={"ID":"0dc7a0fa-fa53-4c98-a342-b3f67b3a1824","Type":"ContainerDied","Data":"f8406bc94f62b89ad46b1c26b1ce28a978b1881a7383fd2523ed439390165a06"}
Nov 01 06:08:37.719331 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:37.719323    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-fd2672-7d58df9c59-2glzq" event={"ID":"0dc7a0fa-fa53-4c98-a342-b3f67b3a1824","Type":"ContainerDied","Data":"23ed5dfcc8e092f9bb32aad9fe48fdb93c5ce384b5577c283b0106ea7caa1cec"}
Nov 01 06:08:37.726546 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:37.726526    5689 scope.go:122] "RemoveContainer" containerID="5cadde8ea56c19d44f37d4191e0e4be3b02ad73dc04cca43cdddffe095d008d4"
Nov 01 06:08:37.726813 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:08:37.726792    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cadde8ea56c19d44f37d4191e0e4be3b02ad73dc04cca43cdddffe095d008d4\": container with ID starting with 5cadde8ea56c19d44f37d4191e0e4be3b02ad73dc04cca43cdddffe095d008d4 not found: ID does not exist" containerID="5cadde8ea56c19d44f37d4191e0e4be3b02ad73dc04cca43cdddffe095d008d4"
Nov 01 06:08:37.726906 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:37.726823    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cadde8ea56c19d44f37d4191e0e4be3b02ad73dc04cca43cdddffe095d008d4"} err="failed to get container status \"5cadde8ea56c19d44f37d4191e0e4be3b02ad73dc04cca43cdddffe095d008d4\": rpc error: code = NotFound desc = could not find container \"5cadde8ea56c19d44f37d4191e0e4be3b02ad73dc04cca43cdddffe095d008d4\": container with ID starting with 5cadde8ea56c19d44f37d4191e0e4be3b02ad73dc04cca43cdddffe095d008d4 not found: ID does not exist"
Nov 01 06:08:37.726906 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:37.726847    5689 scope.go:122] "RemoveContainer" containerID="f8406bc94f62b89ad46b1c26b1ce28a978b1881a7383fd2523ed439390165a06"
Nov 01 06:08:37.736565 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:37.736550    5689 scope.go:122] "RemoveContainer" containerID="f8406bc94f62b89ad46b1c26b1ce28a978b1881a7383fd2523ed439390165a06"
Nov 01 06:08:37.736800 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:08:37.736783    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8406bc94f62b89ad46b1c26b1ce28a978b1881a7383fd2523ed439390165a06\": container with ID starting with f8406bc94f62b89ad46b1c26b1ce28a978b1881a7383fd2523ed439390165a06 not found: ID does not exist" containerID="f8406bc94f62b89ad46b1c26b1ce28a978b1881a7383fd2523ed439390165a06"
Nov 01 06:08:37.736857 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:37.736807    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8406bc94f62b89ad46b1c26b1ce28a978b1881a7383fd2523ed439390165a06"} err="failed to get container status \"f8406bc94f62b89ad46b1c26b1ce28a978b1881a7383fd2523ed439390165a06\": rpc error: code = NotFound desc = could not find container \"f8406bc94f62b89ad46b1c26b1ce28a978b1881a7383fd2523ed439390165a06\": container with ID starting with f8406bc94f62b89ad46b1c26b1ce28a978b1881a7383fd2523ed439390165a06 not found: ID does not exist"
Nov 01 06:08:37.814559 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:37.814528    5689 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dc7a0fa-fa53-4c98-a342-b3f67b3a1824-plugins-config-volume\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:08:37.814559 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:37.814555    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjqmd\" (UniqueName: \"kubernetes.io/projected/0dc7a0fa-fa53-4c98-a342-b3f67b3a1824-kube-api-access-vjqmd\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:08:37.910928 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:37.910896    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-fd2672-7d58df9c59-2glzq"]
Nov 01 06:08:37.912493 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:37.912470    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-fd2672-7d58df9c59-2glzq"]
Nov 01 06:08:38.020903 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:38.020832    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-fd2672-659f849fc9-pqzgj"]
Nov 01 06:08:38.025299 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:38.025274    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-fd2672-659f849fc9-pqzgj"]
Nov 01 06:08:38.067157 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:38.067129    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dc7a0fa-fa53-4c98-a342-b3f67b3a1824" path="/var/lib/kubelet/pods/0dc7a0fa-fa53-4c98-a342-b3f67b3a1824/volumes"
Nov 01 06:08:38.067590 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:38.067571    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75a0b394-e687-407f-8a86-4f2100a6dfc6" path="/var/lib/kubelet/pods/75a0b394-e687-407f-8a86-4f2100a6dfc6/volumes"
Nov 01 06:08:38.714470 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:38.714406    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:08:39.322085 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:39.322054    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:08:41.977293 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:41.977262    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:08:43.958083 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:43.958045    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:08:44.129472 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:44.129414    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:08:45.565919 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:45.565875    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:08:46.018363 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:46.018321    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l"]
Nov 01 06:08:46.018653 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:46.018613    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l" podUID="38e52d68-a35a-4079-a0ec-a7848e909f10" containerName="istio-proxy" containerID="cri-o://fc9f28c29a4c48cf5ad0c48415508ebb81ec43e7cc5496562639e939ac500062" gracePeriod=30
Nov 01 06:08:46.064004 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:46.063973    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:08:46.873131 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:46.873103    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:08:49.359783 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:49.359750    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:08:51.257659 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:51.257637    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l"
Nov 01 06:08:51.334364 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:51.334307    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38e52d68-a35a-4079-a0ec-a7848e909f10-istio-token\" (UniqueName: \"kubernetes.io/projected/38e52d68-a35a-4079-a0ec-a7848e909f10-istio-token\") pod \"38e52d68-a35a-4079-a0ec-a7848e909f10\" (UID: \"38e52d68-a35a-4079-a0ec-a7848e909f10\") "
Nov 01 06:08:51.334364 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:51.334351    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38e52d68-a35a-4079-a0ec-a7848e909f10-kube-api-access-dftcx\" (UniqueName: \"kubernetes.io/projected/38e52d68-a35a-4079-a0ec-a7848e909f10-kube-api-access-dftcx\") pod \"38e52d68-a35a-4079-a0ec-a7848e909f10\" (UID: \"38e52d68-a35a-4079-a0ec-a7848e909f10\") "
Nov 01 06:08:51.334582 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:51.334432    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38e52d68-a35a-4079-a0ec-a7848e909f10-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38e52d68-a35a-4079-a0ec-a7848e909f10-workload-socket\") pod \"38e52d68-a35a-4079-a0ec-a7848e909f10\" (UID: \"38e52d68-a35a-4079-a0ec-a7848e909f10\") "
Nov 01 06:08:51.334582 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:51.334501    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/38e52d68-a35a-4079-a0ec-a7848e909f10-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38e52d68-a35a-4079-a0ec-a7848e909f10-istio-podinfo\") pod \"38e52d68-a35a-4079-a0ec-a7848e909f10\" (UID: \"38e52d68-a35a-4079-a0ec-a7848e909f10\") "
Nov 01 06:08:51.334582 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:51.334532    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38e52d68-a35a-4079-a0ec-a7848e909f10-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38e52d68-a35a-4079-a0ec-a7848e909f10-workload-certs\") pod \"38e52d68-a35a-4079-a0ec-a7848e909f10\" (UID: \"38e52d68-a35a-4079-a0ec-a7848e909f10\") "
Nov 01 06:08:51.334754 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:51.334598    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38e52d68-a35a-4079-a0ec-a7848e909f10-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38e52d68-a35a-4079-a0ec-a7848e909f10-istio-envoy\") pod \"38e52d68-a35a-4079-a0ec-a7848e909f10\" (UID: \"38e52d68-a35a-4079-a0ec-a7848e909f10\") "
Nov 01 06:08:51.334754 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:51.334626    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38e52d68-a35a-4079-a0ec-a7848e909f10-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38e52d68-a35a-4079-a0ec-a7848e909f10-istiod-ca-cert\") pod \"38e52d68-a35a-4079-a0ec-a7848e909f10\" (UID: \"38e52d68-a35a-4079-a0ec-a7848e909f10\") "
Nov 01 06:08:51.334754 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:51.334673    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38e52d68-a35a-4079-a0ec-a7848e909f10-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38e52d68-a35a-4079-a0ec-a7848e909f10-credential-socket\") pod \"38e52d68-a35a-4079-a0ec-a7848e909f10\" (UID: \"38e52d68-a35a-4079-a0ec-a7848e909f10\") "
Nov 01 06:08:51.334754 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:51.334712    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38e52d68-a35a-4079-a0ec-a7848e909f10-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38e52d68-a35a-4079-a0ec-a7848e909f10-istio-data\") pod \"38e52d68-a35a-4079-a0ec-a7848e909f10\" (UID: \"38e52d68-a35a-4079-a0ec-a7848e909f10\") "
Nov 01 06:08:51.334967 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:51.334745    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38e52d68-a35a-4079-a0ec-a7848e909f10-workload-socket" pod "38e52d68-a35a-4079-a0ec-a7848e909f10" (UID: "38e52d68-a35a-4079-a0ec-a7848e909f10"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:08:51.335030 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:51.334962    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38e52d68-a35a-4079-a0ec-a7848e909f10-workload-certs" pod "38e52d68-a35a-4079-a0ec-a7848e909f10" (UID: "38e52d68-a35a-4079-a0ec-a7848e909f10"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:08:51.335030 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:51.334994    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38e52d68-a35a-4079-a0ec-a7848e909f10-credential-socket" pod "38e52d68-a35a-4079-a0ec-a7848e909f10" (UID: "38e52d68-a35a-4079-a0ec-a7848e909f10"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:08:51.335132 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:51.335111    5689 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38e52d68-a35a-4079-a0ec-a7848e909f10-workload-socket\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:08:51.335190 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:51.335154    5689 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38e52d68-a35a-4079-a0ec-a7848e909f10-workload-certs\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:08:51.335190 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:51.335170    5689 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38e52d68-a35a-4079-a0ec-a7848e909f10-credential-socket\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:08:51.335272 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:51.335219    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38e52d68-a35a-4079-a0ec-a7848e909f10-istio-data" pod "38e52d68-a35a-4079-a0ec-a7848e909f10" (UID: "38e52d68-a35a-4079-a0ec-a7848e909f10"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:08:51.335347 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:51.335324    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38e52d68-a35a-4079-a0ec-a7848e909f10-istiod-ca-cert" pod "38e52d68-a35a-4079-a0ec-a7848e909f10" (UID: "38e52d68-a35a-4079-a0ec-a7848e909f10"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:08:51.336931 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:51.336896    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38e52d68-a35a-4079-a0ec-a7848e909f10-istio-envoy" pod "38e52d68-a35a-4079-a0ec-a7848e909f10" (UID: "38e52d68-a35a-4079-a0ec-a7848e909f10"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:08:51.337174 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:51.337151    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/38e52d68-a35a-4079-a0ec-a7848e909f10-istio-podinfo" pod "38e52d68-a35a-4079-a0ec-a7848e909f10" (UID: "38e52d68-a35a-4079-a0ec-a7848e909f10"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:08:51.337280 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:51.337256    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38e52d68-a35a-4079-a0ec-a7848e909f10-istio-token" pod "38e52d68-a35a-4079-a0ec-a7848e909f10" (UID: "38e52d68-a35a-4079-a0ec-a7848e909f10"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:08:51.337514 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:51.337495    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38e52d68-a35a-4079-a0ec-a7848e909f10-kube-api-access-dftcx" pod "38e52d68-a35a-4079-a0ec-a7848e909f10" (UID: "38e52d68-a35a-4079-a0ec-a7848e909f10"). InnerVolumeSpecName "kube-api-access-dftcx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:08:51.436264 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:51.436235    5689 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38e52d68-a35a-4079-a0ec-a7848e909f10-istio-podinfo\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:08:51.436264 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:51.436261    5689 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38e52d68-a35a-4079-a0ec-a7848e909f10-istio-envoy\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:08:51.436264 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:51.436271    5689 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38e52d68-a35a-4079-a0ec-a7848e909f10-istiod-ca-cert\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:08:51.436586 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:51.436279    5689 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38e52d68-a35a-4079-a0ec-a7848e909f10-istio-data\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:08:51.436586 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:51.436289    5689 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38e52d68-a35a-4079-a0ec-a7848e909f10-istio-token\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:08:51.436586 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:51.436300    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dftcx\" (UniqueName: \"kubernetes.io/projected/38e52d68-a35a-4079-a0ec-a7848e909f10-kube-api-access-dftcx\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:08:51.783763 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:51.783728    5689 generic.go:356] "Generic (PLEG): container finished" podID="38e52d68-a35a-4079-a0ec-a7848e909f10" containerID="fc9f28c29a4c48cf5ad0c48415508ebb81ec43e7cc5496562639e939ac500062" exitCode=0
Nov 01 06:08:51.783926 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:51.783778    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l"
Nov 01 06:08:51.783926 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:51.783807    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l" event={"ID":"38e52d68-a35a-4079-a0ec-a7848e909f10","Type":"ContainerDied","Data":"fc9f28c29a4c48cf5ad0c48415508ebb81ec43e7cc5496562639e939ac500062"}
Nov 01 06:08:51.783926 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:51.783877    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l" event={"ID":"38e52d68-a35a-4079-a0ec-a7848e909f10","Type":"ContainerDied","Data":"44a69671a995b105c454b483e7727722b29d1c5bb7ebded765773f5e4585a627"}
Nov 01 06:08:51.783926 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:51.783900    5689 scope.go:122] "RemoveContainer" containerID="fc9f28c29a4c48cf5ad0c48415508ebb81ec43e7cc5496562639e939ac500062"
Nov 01 06:08:51.792994 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:51.792953    5689 scope.go:122] "RemoveContainer" containerID="110ab3735dddf842e0eafe96c19dbdc54d9ce6b8d1c58e60930afd3c1b94de85"
Nov 01 06:08:51.801169 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:51.801152    5689 scope.go:122] "RemoveContainer" containerID="fc9f28c29a4c48cf5ad0c48415508ebb81ec43e7cc5496562639e939ac500062"
Nov 01 06:08:51.801411 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:08:51.801393    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc9f28c29a4c48cf5ad0c48415508ebb81ec43e7cc5496562639e939ac500062\": container with ID starting with fc9f28c29a4c48cf5ad0c48415508ebb81ec43e7cc5496562639e939ac500062 not found: ID does not exist" containerID="fc9f28c29a4c48cf5ad0c48415508ebb81ec43e7cc5496562639e939ac500062"
Nov 01 06:08:51.801480 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:51.801419    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc9f28c29a4c48cf5ad0c48415508ebb81ec43e7cc5496562639e939ac500062"} err="failed to get container status \"fc9f28c29a4c48cf5ad0c48415508ebb81ec43e7cc5496562639e939ac500062\": rpc error: code = NotFound desc = could not find container \"fc9f28c29a4c48cf5ad0c48415508ebb81ec43e7cc5496562639e939ac500062\": container with ID starting with fc9f28c29a4c48cf5ad0c48415508ebb81ec43e7cc5496562639e939ac500062 not found: ID does not exist"
Nov 01 06:08:51.801480 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:51.801436    5689 scope.go:122] "RemoveContainer" containerID="110ab3735dddf842e0eafe96c19dbdc54d9ce6b8d1c58e60930afd3c1b94de85"
Nov 01 06:08:51.801692 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:08:51.801674    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"110ab3735dddf842e0eafe96c19dbdc54d9ce6b8d1c58e60930afd3c1b94de85\": container with ID starting with 110ab3735dddf842e0eafe96c19dbdc54d9ce6b8d1c58e60930afd3c1b94de85 not found: ID does not exist" containerID="110ab3735dddf842e0eafe96c19dbdc54d9ce6b8d1c58e60930afd3c1b94de85"
Nov 01 06:08:51.801752 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:51.801698    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"110ab3735dddf842e0eafe96c19dbdc54d9ce6b8d1c58e60930afd3c1b94de85"} err="failed to get container status \"110ab3735dddf842e0eafe96c19dbdc54d9ce6b8d1c58e60930afd3c1b94de85\": rpc error: code = NotFound desc = could not find container \"110ab3735dddf842e0eafe96c19dbdc54d9ce6b8d1c58e60930afd3c1b94de85\": container with ID starting with 110ab3735dddf842e0eafe96c19dbdc54d9ce6b8d1c58e60930afd3c1b94de85 not found: ID does not exist"
Nov 01 06:08:51.946881 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:51.946852    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l"]
Nov 01 06:08:51.981056 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:51.981030    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-a12c35-openshift-default-84d7454997-whr8l"]
Nov 01 06:08:52.066188 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:52.066117    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38e52d68-a35a-4079-a0ec-a7848e909f10" path="/var/lib/kubelet/pods/38e52d68-a35a-4079-a0ec-a7848e909f10/volumes"
Nov 01 06:08:53.298676 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:53.298647    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:08:53.347610 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:53.347583    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-2fd4c9-7b49b85457-4pklk"]
Nov 01 06:08:53.744500 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:53.744437    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:08:53.917252 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:53.917222    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:08:54.340314 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:54.340283    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:08:58.348118 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:58.348062    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-2fd4c9-7b49b85457-4pklk" podUID="cde94b55-8dd3-426e-aafb-670ec1c4ee7d" containerName="payload-processing" containerID="cri-o://56a1f56199b1f824c2eddd37627c440812f3f69e481bf8cb839ee56895456655" gracePeriod=25
Nov 01 06:08:58.673210 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:58.673183    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-2fd4c9-6674594479-74lqf"]
Nov 01 06:08:58.673408 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:58.673376    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2fd4c9-6674594479-74lqf" podUID="9fe0ed2a-6b37-4a3c-bcaf-21afae50cafb" containerName="maas-api" containerID="cri-o://8d5493454e66dbf81e30cabe5c4d3b0edc5be13c82b7d86c6483f7d6ced1b769" gracePeriod=30
Nov 01 06:08:58.816159 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:58.816124    5689 generic.go:356] "Generic (PLEG): container finished" podID="9fe0ed2a-6b37-4a3c-bcaf-21afae50cafb" containerID="8d5493454e66dbf81e30cabe5c4d3b0edc5be13c82b7d86c6483f7d6ced1b769" exitCode=0
Nov 01 06:08:58.816311 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:58.816171    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2fd4c9-6674594479-74lqf" event={"ID":"9fe0ed2a-6b37-4a3c-bcaf-21afae50cafb","Type":"ContainerDied","Data":"8d5493454e66dbf81e30cabe5c4d3b0edc5be13c82b7d86c6483f7d6ced1b769"}
Nov 01 06:08:58.830110 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:58.830064    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:08:58.922984 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:58.922961    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2fd4c9-6674594479-74lqf"
Nov 01 06:08:59.004263 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:59.004205    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fe0ed2a-6b37-4a3c-bcaf-21afae50cafb-kube-api-access-dq84z\" (UniqueName: \"kubernetes.io/projected/9fe0ed2a-6b37-4a3c-bcaf-21afae50cafb-kube-api-access-dq84z\") pod \"9fe0ed2a-6b37-4a3c-bcaf-21afae50cafb\" (UID: \"9fe0ed2a-6b37-4a3c-bcaf-21afae50cafb\") "
Nov 01 06:08:59.004263 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:59.004238    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fe0ed2a-6b37-4a3c-bcaf-21afae50cafb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fe0ed2a-6b37-4a3c-bcaf-21afae50cafb-metrics-certs\") pod \"9fe0ed2a-6b37-4a3c-bcaf-21afae50cafb\" (UID: \"9fe0ed2a-6b37-4a3c-bcaf-21afae50cafb\") "
Nov 01 06:08:59.004446 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:59.004266    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fe0ed2a-6b37-4a3c-bcaf-21afae50cafb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fe0ed2a-6b37-4a3c-bcaf-21afae50cafb-maas-api-tls\") pod \"9fe0ed2a-6b37-4a3c-bcaf-21afae50cafb\" (UID: \"9fe0ed2a-6b37-4a3c-bcaf-21afae50cafb\") "
Nov 01 06:08:59.004446 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:59.004292    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fe0ed2a-6b37-4a3c-bcaf-21afae50cafb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fe0ed2a-6b37-4a3c-bcaf-21afae50cafb-openshift-service-ca\") pod \"9fe0ed2a-6b37-4a3c-bcaf-21afae50cafb\" (UID: \"9fe0ed2a-6b37-4a3c-bcaf-21afae50cafb\") "
Nov 01 06:08:59.004792 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:59.004767    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fe0ed2a-6b37-4a3c-bcaf-21afae50cafb-openshift-service-ca" pod "9fe0ed2a-6b37-4a3c-bcaf-21afae50cafb" (UID: "9fe0ed2a-6b37-4a3c-bcaf-21afae50cafb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:08:59.006432 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:59.006412    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fe0ed2a-6b37-4a3c-bcaf-21afae50cafb-kube-api-access-dq84z" pod "9fe0ed2a-6b37-4a3c-bcaf-21afae50cafb" (UID: "9fe0ed2a-6b37-4a3c-bcaf-21afae50cafb"). InnerVolumeSpecName "kube-api-access-dq84z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:08:59.006541 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:59.006447    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fe0ed2a-6b37-4a3c-bcaf-21afae50cafb-maas-api-tls" pod "9fe0ed2a-6b37-4a3c-bcaf-21afae50cafb" (UID: "9fe0ed2a-6b37-4a3c-bcaf-21afae50cafb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:08:59.006594 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:59.006550    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fe0ed2a-6b37-4a3c-bcaf-21afae50cafb-metrics-certs" pod "9fe0ed2a-6b37-4a3c-bcaf-21afae50cafb" (UID: "9fe0ed2a-6b37-4a3c-bcaf-21afae50cafb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:08:59.105964 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:59.105928    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dq84z\" (UniqueName: \"kubernetes.io/projected/9fe0ed2a-6b37-4a3c-bcaf-21afae50cafb-kube-api-access-dq84z\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:08:59.105964 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:59.105958    5689 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fe0ed2a-6b37-4a3c-bcaf-21afae50cafb-metrics-certs\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:08:59.105964 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:59.105969    5689 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fe0ed2a-6b37-4a3c-bcaf-21afae50cafb-maas-api-tls\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:08:59.106196 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:59.105978    5689 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fe0ed2a-6b37-4a3c-bcaf-21afae50cafb-openshift-service-ca\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:08:59.371220 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:59.371136    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-2fd4c9-7b49b85457-4pklk" podUID="cde94b55-8dd3-426e-aafb-670ec1c4ee7d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:08:59.374757 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:59.374732    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:08:59.822073 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:59.822044    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2fd4c9-6674594479-74lqf"
Nov 01 06:08:59.822275 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:59.822073    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2fd4c9-6674594479-74lqf" event={"ID":"9fe0ed2a-6b37-4a3c-bcaf-21afae50cafb","Type":"ContainerDied","Data":"3088623c340b3fd5b3c28fc1a308835c714bb2a12679875a5bf7128da5207f31"}
Nov 01 06:08:59.822275 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:59.822130    5689 scope.go:122] "RemoveContainer" containerID="8d5493454e66dbf81e30cabe5c4d3b0edc5be13c82b7d86c6483f7d6ced1b769"
Nov 01 06:08:59.985157 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:08:59.985131    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-2fd4c9-6674594479-74lqf"]
Nov 01 06:09:00.034310 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:00.034280    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-2fd4c9-6674594479-74lqf"]
Nov 01 06:09:00.067654 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:09:00.067617    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 06:09:00.068919 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:00.068897    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fe0ed2a-6b37-4a3c-bcaf-21afae50cafb" path="/var/lib/kubelet/pods/9fe0ed2a-6b37-4a3c-bcaf-21afae50cafb/volumes"
Nov 01 06:09:03.170883 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:03.170849    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:09:05.209593 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:05.209563    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:09:05.395023 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:05.394989    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:09:06.773183 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:06.773145    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:09:07.218228 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:07.218183    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8"]
Nov 01 06:09:07.218400 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:07.218383    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8" podUID="82550158-2a2b-4dd6-b8e7-8a777954dfd6" containerName="istio-proxy" containerID="cri-o://09e84abd09aae2ef6adb280141f2c35b9f92034c18c685d8c187436c6b424ba7" gracePeriod=30
Nov 01 06:09:07.255037 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:07.255004    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:09:08.528720 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:08.528691    5689 patch_prober.go:29] interesting pod/e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 06:09:08.529101 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:08.528739    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8" podUID="82550158-2a2b-4dd6-b8e7-8a777954dfd6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 06:09:09.371657 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:09.371618    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-2fd4c9-7b49b85457-4pklk" podUID="cde94b55-8dd3-426e-aafb-670ec1c4ee7d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:09:09.934201 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:09.934136    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:09:10.116049 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:10.116000    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:09:10.949911 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:10.949875    5689 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-dc89h_817d6a15-8f0e-406f-8dd2-bed3b6323796/limitador/0.log"
Nov 01 06:09:12.419263 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:12.419222    5689 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-hstph_4959df17-a373-4cef-b555-e342fb4ebbdf/manager/0.log"
Nov 01 06:09:12.486805 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:12.486747    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8"
Nov 01 06:09:12.527048 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:12.527017    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82550158-2a2b-4dd6-b8e7-8a777954dfd6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82550158-2a2b-4dd6-b8e7-8a777954dfd6-istio-data\") pod \"82550158-2a2b-4dd6-b8e7-8a777954dfd6\" (UID: \"82550158-2a2b-4dd6-b8e7-8a777954dfd6\") "
Nov 01 06:09:12.527173 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:12.527072    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/82550158-2a2b-4dd6-b8e7-8a777954dfd6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82550158-2a2b-4dd6-b8e7-8a777954dfd6-istio-podinfo\") pod \"82550158-2a2b-4dd6-b8e7-8a777954dfd6\" (UID: \"82550158-2a2b-4dd6-b8e7-8a777954dfd6\") "
Nov 01 06:09:12.527173 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:12.527092    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82550158-2a2b-4dd6-b8e7-8a777954dfd6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82550158-2a2b-4dd6-b8e7-8a777954dfd6-istiod-ca-cert\") pod \"82550158-2a2b-4dd6-b8e7-8a777954dfd6\" (UID: \"82550158-2a2b-4dd6-b8e7-8a777954dfd6\") "
Nov 01 06:09:12.527173 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:12.527140    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82550158-2a2b-4dd6-b8e7-8a777954dfd6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82550158-2a2b-4dd6-b8e7-8a777954dfd6-istio-envoy\") pod \"82550158-2a2b-4dd6-b8e7-8a777954dfd6\" (UID: \"82550158-2a2b-4dd6-b8e7-8a777954dfd6\") "
Nov 01 06:09:12.527350 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:12.527223    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82550158-2a2b-4dd6-b8e7-8a777954dfd6-kube-api-access-9r5rn\" (UniqueName: \"kubernetes.io/projected/82550158-2a2b-4dd6-b8e7-8a777954dfd6-kube-api-access-9r5rn\") pod \"82550158-2a2b-4dd6-b8e7-8a777954dfd6\" (UID: \"82550158-2a2b-4dd6-b8e7-8a777954dfd6\") "
Nov 01 06:09:12.527350 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:12.527279    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82550158-2a2b-4dd6-b8e7-8a777954dfd6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82550158-2a2b-4dd6-b8e7-8a777954dfd6-workload-socket\") pod \"82550158-2a2b-4dd6-b8e7-8a777954dfd6\" (UID: \"82550158-2a2b-4dd6-b8e7-8a777954dfd6\") "
Nov 01 06:09:12.527350 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:12.527291    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82550158-2a2b-4dd6-b8e7-8a777954dfd6-istio-data" pod "82550158-2a2b-4dd6-b8e7-8a777954dfd6" (UID: "82550158-2a2b-4dd6-b8e7-8a777954dfd6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:09:12.527350 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:12.527329    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82550158-2a2b-4dd6-b8e7-8a777954dfd6-istio-token\" (UniqueName: \"kubernetes.io/projected/82550158-2a2b-4dd6-b8e7-8a777954dfd6-istio-token\") pod \"82550158-2a2b-4dd6-b8e7-8a777954dfd6\" (UID: \"82550158-2a2b-4dd6-b8e7-8a777954dfd6\") "
Nov 01 06:09:12.527607 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:12.527371    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82550158-2a2b-4dd6-b8e7-8a777954dfd6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82550158-2a2b-4dd6-b8e7-8a777954dfd6-workload-certs\") pod \"82550158-2a2b-4dd6-b8e7-8a777954dfd6\" (UID: \"82550158-2a2b-4dd6-b8e7-8a777954dfd6\") "
Nov 01 06:09:12.527607 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:12.527407    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82550158-2a2b-4dd6-b8e7-8a777954dfd6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82550158-2a2b-4dd6-b8e7-8a777954dfd6-credential-socket\") pod \"82550158-2a2b-4dd6-b8e7-8a777954dfd6\" (UID: \"82550158-2a2b-4dd6-b8e7-8a777954dfd6\") "
Nov 01 06:09:12.527607 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:12.527558    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82550158-2a2b-4dd6-b8e7-8a777954dfd6-istiod-ca-cert" pod "82550158-2a2b-4dd6-b8e7-8a777954dfd6" (UID: "82550158-2a2b-4dd6-b8e7-8a777954dfd6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:09:12.527607 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:12.527568    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82550158-2a2b-4dd6-b8e7-8a777954dfd6-workload-socket" pod "82550158-2a2b-4dd6-b8e7-8a777954dfd6" (UID: "82550158-2a2b-4dd6-b8e7-8a777954dfd6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:09:12.527836 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:12.527749    5689 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82550158-2a2b-4dd6-b8e7-8a777954dfd6-workload-socket\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:09:12.527836 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:12.527771    5689 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82550158-2a2b-4dd6-b8e7-8a777954dfd6-istio-data\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:09:12.527836 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:12.527786    5689 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82550158-2a2b-4dd6-b8e7-8a777954dfd6-istiod-ca-cert\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:09:12.527836 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:12.527769    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82550158-2a2b-4dd6-b8e7-8a777954dfd6-credential-socket" pod "82550158-2a2b-4dd6-b8e7-8a777954dfd6" (UID: "82550158-2a2b-4dd6-b8e7-8a777954dfd6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:09:12.527836 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:12.527806    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82550158-2a2b-4dd6-b8e7-8a777954dfd6-workload-certs" pod "82550158-2a2b-4dd6-b8e7-8a777954dfd6" (UID: "82550158-2a2b-4dd6-b8e7-8a777954dfd6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:09:12.529517 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:12.529481    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82550158-2a2b-4dd6-b8e7-8a777954dfd6-kube-api-access-9r5rn" pod "82550158-2a2b-4dd6-b8e7-8a777954dfd6" (UID: "82550158-2a2b-4dd6-b8e7-8a777954dfd6"). InnerVolumeSpecName "kube-api-access-9r5rn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:09:12.529641 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:12.529537    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82550158-2a2b-4dd6-b8e7-8a777954dfd6-istio-envoy" pod "82550158-2a2b-4dd6-b8e7-8a777954dfd6" (UID: "82550158-2a2b-4dd6-b8e7-8a777954dfd6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:09:12.529710 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:12.529658    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/82550158-2a2b-4dd6-b8e7-8a777954dfd6-istio-podinfo" pod "82550158-2a2b-4dd6-b8e7-8a777954dfd6" (UID: "82550158-2a2b-4dd6-b8e7-8a777954dfd6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:09:12.529969 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:12.529947    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82550158-2a2b-4dd6-b8e7-8a777954dfd6-istio-token" pod "82550158-2a2b-4dd6-b8e7-8a777954dfd6" (UID: "82550158-2a2b-4dd6-b8e7-8a777954dfd6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:09:12.629184 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:12.629146    5689 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82550158-2a2b-4dd6-b8e7-8a777954dfd6-istio-envoy\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:09:12.629184 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:12.629182    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9r5rn\" (UniqueName: \"kubernetes.io/projected/82550158-2a2b-4dd6-b8e7-8a777954dfd6-kube-api-access-9r5rn\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:09:12.629184 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:12.629193    5689 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82550158-2a2b-4dd6-b8e7-8a777954dfd6-istio-token\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:09:12.629408 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:12.629202    5689 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82550158-2a2b-4dd6-b8e7-8a777954dfd6-workload-certs\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:09:12.629408 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:12.629210    5689 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82550158-2a2b-4dd6-b8e7-8a777954dfd6-credential-socket\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:09:12.629408 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:12.629219    5689 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82550158-2a2b-4dd6-b8e7-8a777954dfd6-istio-podinfo\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:09:12.880927 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:12.880851    5689 generic.go:356] "Generic (PLEG): container finished" podID="82550158-2a2b-4dd6-b8e7-8a777954dfd6" containerID="09e84abd09aae2ef6adb280141f2c35b9f92034c18c685d8c187436c6b424ba7" exitCode=0
Nov 01 06:09:12.880927 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:12.880885    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8" event={"ID":"82550158-2a2b-4dd6-b8e7-8a777954dfd6","Type":"ContainerDied","Data":"09e84abd09aae2ef6adb280141f2c35b9f92034c18c685d8c187436c6b424ba7"}
Nov 01 06:09:12.880927 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:12.880903    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8"
Nov 01 06:09:12.881148 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:12.880952    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8" event={"ID":"82550158-2a2b-4dd6-b8e7-8a777954dfd6","Type":"ContainerDied","Data":"d4ce87b1a64dc2a740d826d1442e214f435a78437a81c03db5a8e08461285eb6"}
Nov 01 06:09:12.881148 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:12.880974    5689 scope.go:122] "RemoveContainer" containerID="09e84abd09aae2ef6adb280141f2c35b9f92034c18c685d8c187436c6b424ba7"
Nov 01 06:09:12.890079 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:12.890060    5689 scope.go:122] "RemoveContainer" containerID="68b67a65e0852248333ebf340d765a36b08644b323b5e99575a2a8409bc5715b"
Nov 01 06:09:12.897579 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:12.897564    5689 scope.go:122] "RemoveContainer" containerID="09e84abd09aae2ef6adb280141f2c35b9f92034c18c685d8c187436c6b424ba7"
Nov 01 06:09:12.897807 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:09:12.897786    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09e84abd09aae2ef6adb280141f2c35b9f92034c18c685d8c187436c6b424ba7\": container with ID starting with 09e84abd09aae2ef6adb280141f2c35b9f92034c18c685d8c187436c6b424ba7 not found: ID does not exist" containerID="09e84abd09aae2ef6adb280141f2c35b9f92034c18c685d8c187436c6b424ba7"
Nov 01 06:09:12.897869 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:12.897813    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09e84abd09aae2ef6adb280141f2c35b9f92034c18c685d8c187436c6b424ba7"} err="failed to get container status \"09e84abd09aae2ef6adb280141f2c35b9f92034c18c685d8c187436c6b424ba7\": rpc error: code = NotFound desc = could not find container \"09e84abd09aae2ef6adb280141f2c35b9f92034c18c685d8c187436c6b424ba7\": container with ID starting with 09e84abd09aae2ef6adb280141f2c35b9f92034c18c685d8c187436c6b424ba7 not found: ID does not exist"
Nov 01 06:09:12.897869 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:12.897829    5689 scope.go:122] "RemoveContainer" containerID="68b67a65e0852248333ebf340d765a36b08644b323b5e99575a2a8409bc5715b"
Nov 01 06:09:12.898066 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:09:12.898051    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68b67a65e0852248333ebf340d765a36b08644b323b5e99575a2a8409bc5715b\": container with ID starting with 68b67a65e0852248333ebf340d765a36b08644b323b5e99575a2a8409bc5715b not found: ID does not exist" containerID="68b67a65e0852248333ebf340d765a36b08644b323b5e99575a2a8409bc5715b"
Nov 01 06:09:12.898108 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:12.898070    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68b67a65e0852248333ebf340d765a36b08644b323b5e99575a2a8409bc5715b"} err="failed to get container status \"68b67a65e0852248333ebf340d765a36b08644b323b5e99575a2a8409bc5715b\": rpc error: code = NotFound desc = could not find container \"68b67a65e0852248333ebf340d765a36b08644b323b5e99575a2a8409bc5715b\": container with ID starting with 68b67a65e0852248333ebf340d765a36b08644b323b5e99575a2a8409bc5715b not found: ID does not exist"
Nov 01 06:09:13.045104 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:13.045078    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8"]
Nov 01 06:09:13.049691 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:13.049666    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-2fd4c9-openshift-default-69b7b9b68f-xdmp8"]
Nov 01 06:09:13.967087 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:13.967049    5689 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-pjms7_ff4e020b-0316-4b29-a761-904818e3fb4f/manager/0.log"
Nov 01 06:09:14.066018 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:14.065985    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82550158-2a2b-4dd6-b8e7-8a777954dfd6" path="/var/lib/kubelet/pods/82550158-2a2b-4dd6-b8e7-8a777954dfd6/volumes"
Nov 01 06:09:16.112468 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:16.112427    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:09:18.586831 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:18.586807    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-2fd4c9-7b49b85457-4pklk"
Nov 01 06:09:18.680360 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:18.680338    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cde94b55-8dd3-426e-aafb-670ec1c4ee7d-kube-api-access-hzzgh\" (UniqueName: \"kubernetes.io/projected/cde94b55-8dd3-426e-aafb-670ec1c4ee7d-kube-api-access-hzzgh\") pod \"cde94b55-8dd3-426e-aafb-670ec1c4ee7d\" (UID: \"cde94b55-8dd3-426e-aafb-670ec1c4ee7d\") "
Nov 01 06:09:18.680555 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:18.680535    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cde94b55-8dd3-426e-aafb-670ec1c4ee7d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cde94b55-8dd3-426e-aafb-670ec1c4ee7d-plugins-config-volume\") pod \"cde94b55-8dd3-426e-aafb-670ec1c4ee7d\" (UID: \"cde94b55-8dd3-426e-aafb-670ec1c4ee7d\") "
Nov 01 06:09:18.680865 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:18.680847    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cde94b55-8dd3-426e-aafb-670ec1c4ee7d-plugins-config-volume" pod "cde94b55-8dd3-426e-aafb-670ec1c4ee7d" (UID: "cde94b55-8dd3-426e-aafb-670ec1c4ee7d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:09:18.682486 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:18.682447    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cde94b55-8dd3-426e-aafb-670ec1c4ee7d-kube-api-access-hzzgh" pod "cde94b55-8dd3-426e-aafb-670ec1c4ee7d" (UID: "cde94b55-8dd3-426e-aafb-670ec1c4ee7d"). InnerVolumeSpecName "kube-api-access-hzzgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:09:18.781798 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:18.781767    5689 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cde94b55-8dd3-426e-aafb-670ec1c4ee7d-plugins-config-volume\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:09:18.781798 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:18.781794    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzzgh\" (UniqueName: \"kubernetes.io/projected/cde94b55-8dd3-426e-aafb-670ec1c4ee7d-kube-api-access-hzzgh\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:09:18.910464 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:18.910428    5689 generic.go:356] "Generic (PLEG): container finished" podID="cde94b55-8dd3-426e-aafb-670ec1c4ee7d" containerID="56a1f56199b1f824c2eddd37627c440812f3f69e481bf8cb839ee56895456655" exitCode=0
Nov 01 06:09:18.910621 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:18.910487    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-2fd4c9-7b49b85457-4pklk"
Nov 01 06:09:18.910621 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:18.910497    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-2fd4c9-7b49b85457-4pklk" event={"ID":"cde94b55-8dd3-426e-aafb-670ec1c4ee7d","Type":"ContainerDied","Data":"56a1f56199b1f824c2eddd37627c440812f3f69e481bf8cb839ee56895456655"}
Nov 01 06:09:18.910621 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:18.910537    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-2fd4c9-7b49b85457-4pklk" event={"ID":"cde94b55-8dd3-426e-aafb-670ec1c4ee7d","Type":"ContainerDied","Data":"aa26aed818575c331027e5e16d087444ede2ea43850b4f6484a10eb95f61fd54"}
Nov 01 06:09:18.910621 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:18.910556    5689 scope.go:122] "RemoveContainer" containerID="56a1f56199b1f824c2eddd37627c440812f3f69e481bf8cb839ee56895456655"
Nov 01 06:09:18.919675 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:18.919654    5689 scope.go:122] "RemoveContainer" containerID="56a1f56199b1f824c2eddd37627c440812f3f69e481bf8cb839ee56895456655"
Nov 01 06:09:18.919935 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:09:18.919913    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56a1f56199b1f824c2eddd37627c440812f3f69e481bf8cb839ee56895456655\": container with ID starting with 56a1f56199b1f824c2eddd37627c440812f3f69e481bf8cb839ee56895456655 not found: ID does not exist" containerID="56a1f56199b1f824c2eddd37627c440812f3f69e481bf8cb839ee56895456655"
Nov 01 06:09:18.920006 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:18.919949    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56a1f56199b1f824c2eddd37627c440812f3f69e481bf8cb839ee56895456655"} err="failed to get container status \"56a1f56199b1f824c2eddd37627c440812f3f69e481bf8cb839ee56895456655\": rpc error: code = NotFound desc = could not find container \"56a1f56199b1f824c2eddd37627c440812f3f69e481bf8cb839ee56895456655\": container with ID starting with 56a1f56199b1f824c2eddd37627c440812f3f69e481bf8cb839ee56895456655 not found: ID does not exist"
Nov 01 06:09:19.074540 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:19.074436    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-2fd4c9-7b49b85457-4pklk"]
Nov 01 06:09:19.080406 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:19.080382    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-2fd4c9-7b49b85457-4pklk"]
Nov 01 06:09:19.176976 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:19.176949    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:09:19.897797 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:19.897768    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-3951eb77-openshift-default-85859ff58c-gf998"]
Nov 01 06:09:19.898175 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:19.898120    5689 cpu_manager.go:414] "containerMap: removing container" podUID="3d1f1368-d8ef-480f-916c-27fb53b44b25" containerName="maas-api"
Nov 01 06:09:19.898175 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:19.898132    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d1f1368-d8ef-480f-916c-27fb53b44b25" containerName="maas-api"
Nov 01 06:09:19.898175 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:19.898140    5689 cpu_manager.go:414] "containerMap: removing container" podUID="b7183c3f-d319-4c01-83fe-0df57b41b69c" containerName="istio-proxy"
Nov 01 06:09:19.898175 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:19.898145    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7183c3f-d319-4c01-83fe-0df57b41b69c" containerName="istio-proxy"
Nov 01 06:09:19.898175 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:19.898159    5689 cpu_manager.go:414] "containerMap: removing container" podUID="38e52d68-a35a-4079-a0ec-a7848e909f10" containerName="istio-proxy"
Nov 01 06:09:19.898175 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:19.898164    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38e52d68-a35a-4079-a0ec-a7848e909f10" containerName="istio-proxy"
Nov 01 06:09:19.898175 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:19.898175    5689 cpu_manager.go:414] "containerMap: removing container" podUID="82550158-2a2b-4dd6-b8e7-8a777954dfd6" containerName="istio-proxy"
Nov 01 06:09:19.898404 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:19.898180    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82550158-2a2b-4dd6-b8e7-8a777954dfd6" containerName="istio-proxy"
Nov 01 06:09:19.898404 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:19.898187    5689 cpu_manager.go:414] "containerMap: removing container" podUID="451c733b-f9b7-4bc0-a113-a4e7607bba9f" containerName="main"
Nov 01 06:09:19.898404 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:19.898192    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="451c733b-f9b7-4bc0-a113-a4e7607bba9f" containerName="main"
Nov 01 06:09:19.898404 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:19.898203    5689 cpu_manager.go:414] "containerMap: removing container" podUID="38e52d68-a35a-4079-a0ec-a7848e909f10" containerName="istio-proxy"
Nov 01 06:09:19.898404 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:19.898208    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38e52d68-a35a-4079-a0ec-a7848e909f10" containerName="istio-proxy"
Nov 01 06:09:19.898404 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:19.898218    5689 cpu_manager.go:414] "containerMap: removing container" podUID="9fe0ed2a-6b37-4a3c-bcaf-21afae50cafb" containerName="maas-api"
Nov 01 06:09:19.898404 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:19.898222    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fe0ed2a-6b37-4a3c-bcaf-21afae50cafb" containerName="maas-api"
Nov 01 06:09:19.898404 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:19.898231    5689 cpu_manager.go:414] "containerMap: removing container" podUID="f9e983e7-105f-4706-8872-3b3893f748f0" containerName="maas-api"
Nov 01 06:09:19.898404 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:19.898235    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9e983e7-105f-4706-8872-3b3893f748f0" containerName="maas-api"
Nov 01 06:09:19.898404 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:19.898246    5689 cpu_manager.go:414] "containerMap: removing container" podUID="82550158-2a2b-4dd6-b8e7-8a777954dfd6" containerName="istio-proxy"
Nov 01 06:09:19.898404 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:19.898250    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82550158-2a2b-4dd6-b8e7-8a777954dfd6" containerName="istio-proxy"
Nov 01 06:09:19.898404 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:19.898257    5689 cpu_manager.go:414] "containerMap: removing container" podUID="75a0b394-e687-407f-8a86-4f2100a6dfc6" containerName="payload-processing"
Nov 01 06:09:19.898404 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:19.898261    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75a0b394-e687-407f-8a86-4f2100a6dfc6" containerName="payload-processing"
Nov 01 06:09:19.898404 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:19.898270    5689 cpu_manager.go:414] "containerMap: removing container" podUID="0dc7a0fa-fa53-4c98-a342-b3f67b3a1824" containerName="payload-pre-processing"
Nov 01 06:09:19.898404 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:19.898274    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dc7a0fa-fa53-4c98-a342-b3f67b3a1824" containerName="payload-pre-processing"
Nov 01 06:09:19.898404 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:19.898283    5689 cpu_manager.go:414] "containerMap: removing container" podUID="cde94b55-8dd3-426e-aafb-670ec1c4ee7d" containerName="payload-processing"
Nov 01 06:09:19.898404 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:19.898287    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cde94b55-8dd3-426e-aafb-670ec1c4ee7d" containerName="payload-processing"
Nov 01 06:09:19.898404 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:19.898293    5689 cpu_manager.go:414] "containerMap: removing container" podUID="9a008fc6-3439-4701-8748-c0d0928c0f8d" containerName="istio-proxy"
Nov 01 06:09:19.898404 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:19.898297    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a008fc6-3439-4701-8748-c0d0928c0f8d" containerName="istio-proxy"
Nov 01 06:09:19.898404 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:19.898348    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7183c3f-d319-4c01-83fe-0df57b41b69c" containerName="istio-proxy"
Nov 01 06:09:19.898404 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:19.898357    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dc7a0fa-fa53-4c98-a342-b3f67b3a1824" containerName="payload-pre-processing"
Nov 01 06:09:19.898404 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:19.898363    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="cde94b55-8dd3-426e-aafb-670ec1c4ee7d" containerName="payload-processing"
Nov 01 06:09:19.898404 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:19.898368    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a008fc6-3439-4701-8748-c0d0928c0f8d" containerName="istio-proxy"
Nov 01 06:09:19.898404 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:19.898374    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d1f1368-d8ef-480f-916c-27fb53b44b25" containerName="maas-api"
Nov 01 06:09:19.898404 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:19.898380    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="38e52d68-a35a-4079-a0ec-a7848e909f10" containerName="istio-proxy"
Nov 01 06:09:19.898404 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:19.898385    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="82550158-2a2b-4dd6-b8e7-8a777954dfd6" containerName="istio-proxy"
Nov 01 06:09:19.898404 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:19.898390    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="451c733b-f9b7-4bc0-a113-a4e7607bba9f" containerName="main"
Nov 01 06:09:19.898404 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:19.898396    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="38e52d68-a35a-4079-a0ec-a7848e909f10" containerName="istio-proxy"
Nov 01 06:09:19.898404 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:19.898403    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fe0ed2a-6b37-4a3c-bcaf-21afae50cafb" containerName="maas-api"
Nov 01 06:09:19.898404 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:19.898408    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9e983e7-105f-4706-8872-3b3893f748f0" containerName="maas-api"
Nov 01 06:09:19.898404 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:19.898414    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="82550158-2a2b-4dd6-b8e7-8a777954dfd6" containerName="istio-proxy"
Nov 01 06:09:19.898404 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:19.898418    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="75a0b394-e687-407f-8a86-4f2100a6dfc6" containerName="payload-processing"
Nov 01 06:09:19.902664 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:19.902647    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3951eb77-openshift-default-85859ff58c-gf998"
Nov 01 06:09:19.964103 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:19.964071    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3951eb77-openshift-default-85859ff58c-gf998"]
Nov 01 06:09:19.992067 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:19.992038    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60bdc2d2-b519-4565-8375-61a5254e4f54-istiod-ca-cert\") pod \"e2e-ait-3951eb77-openshift-default-85859ff58c-gf998\" (UID: \"60bdc2d2-b519-4565-8375-61a5254e4f54\") " pod="openshift-ingress/e2e-ait-3951eb77-openshift-default-85859ff58c-gf998"
Nov 01 06:09:19.992237 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:19.992072    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj46x\" (UniqueName: \"kubernetes.io/projected/60bdc2d2-b519-4565-8375-61a5254e4f54-kube-api-access-pj46x\") pod \"e2e-ait-3951eb77-openshift-default-85859ff58c-gf998\" (UID: \"60bdc2d2-b519-4565-8375-61a5254e4f54\") " pod="openshift-ingress/e2e-ait-3951eb77-openshift-default-85859ff58c-gf998"
Nov 01 06:09:19.992237 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:19.992106    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60bdc2d2-b519-4565-8375-61a5254e4f54-istio-podinfo\") pod \"e2e-ait-3951eb77-openshift-default-85859ff58c-gf998\" (UID: \"60bdc2d2-b519-4565-8375-61a5254e4f54\") " pod="openshift-ingress/e2e-ait-3951eb77-openshift-default-85859ff58c-gf998"
Nov 01 06:09:19.992335 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:19.992261    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60bdc2d2-b519-4565-8375-61a5254e4f54-workload-socket\") pod \"e2e-ait-3951eb77-openshift-default-85859ff58c-gf998\" (UID: \"60bdc2d2-b519-4565-8375-61a5254e4f54\") " pod="openshift-ingress/e2e-ait-3951eb77-openshift-default-85859ff58c-gf998"
Nov 01 06:09:19.992335 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:19.992308    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60bdc2d2-b519-4565-8375-61a5254e4f54-workload-certs\") pod \"e2e-ait-3951eb77-openshift-default-85859ff58c-gf998\" (UID: \"60bdc2d2-b519-4565-8375-61a5254e4f54\") " pod="openshift-ingress/e2e-ait-3951eb77-openshift-default-85859ff58c-gf998"
Nov 01 06:09:19.992445 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:19.992412    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60bdc2d2-b519-4565-8375-61a5254e4f54-istio-data\") pod \"e2e-ait-3951eb77-openshift-default-85859ff58c-gf998\" (UID: \"60bdc2d2-b519-4565-8375-61a5254e4f54\") " pod="openshift-ingress/e2e-ait-3951eb77-openshift-default-85859ff58c-gf998"
Nov 01 06:09:19.992524 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:19.992466    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60bdc2d2-b519-4565-8375-61a5254e4f54-credential-socket\") pod \"e2e-ait-3951eb77-openshift-default-85859ff58c-gf998\" (UID: \"60bdc2d2-b519-4565-8375-61a5254e4f54\") " pod="openshift-ingress/e2e-ait-3951eb77-openshift-default-85859ff58c-gf998"
Nov 01 06:09:19.992562 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:19.992523    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60bdc2d2-b519-4565-8375-61a5254e4f54-istio-envoy\") pod \"e2e-ait-3951eb77-openshift-default-85859ff58c-gf998\" (UID: \"60bdc2d2-b519-4565-8375-61a5254e4f54\") " pod="openshift-ingress/e2e-ait-3951eb77-openshift-default-85859ff58c-gf998"
Nov 01 06:09:19.992562 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:19.992548    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60bdc2d2-b519-4565-8375-61a5254e4f54-istio-token\") pod \"e2e-ait-3951eb77-openshift-default-85859ff58c-gf998\" (UID: \"60bdc2d2-b519-4565-8375-61a5254e4f54\") " pod="openshift-ingress/e2e-ait-3951eb77-openshift-default-85859ff58c-gf998"
Nov 01 06:09:20.011094 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:20.011077    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-3951eb77-openshift-default-dockercfg-694bx\""
Nov 01 06:09:20.022919 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:20.022892    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:09:20.066625 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:20.066596    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cde94b55-8dd3-426e-aafb-670ec1c4ee7d" path="/var/lib/kubelet/pods/cde94b55-8dd3-426e-aafb-670ec1c4ee7d/volumes"
Nov 01 06:09:20.092993 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:20.092972    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60bdc2d2-b519-4565-8375-61a5254e4f54-istio-envoy\") pod \"e2e-ait-3951eb77-openshift-default-85859ff58c-gf998\" (UID: \"60bdc2d2-b519-4565-8375-61a5254e4f54\") " pod="openshift-ingress/e2e-ait-3951eb77-openshift-default-85859ff58c-gf998"
Nov 01 06:09:20.093112 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:20.093003    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60bdc2d2-b519-4565-8375-61a5254e4f54-istio-token\") pod \"e2e-ait-3951eb77-openshift-default-85859ff58c-gf998\" (UID: \"60bdc2d2-b519-4565-8375-61a5254e4f54\") " pod="openshift-ingress/e2e-ait-3951eb77-openshift-default-85859ff58c-gf998"
Nov 01 06:09:20.093112 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:20.093023    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60bdc2d2-b519-4565-8375-61a5254e4f54-istiod-ca-cert\") pod \"e2e-ait-3951eb77-openshift-default-85859ff58c-gf998\" (UID: \"60bdc2d2-b519-4565-8375-61a5254e4f54\") " pod="openshift-ingress/e2e-ait-3951eb77-openshift-default-85859ff58c-gf998"
Nov 01 06:09:20.093112 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:20.093076    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj46x\" (UniqueName: \"kubernetes.io/projected/60bdc2d2-b519-4565-8375-61a5254e4f54-kube-api-access-pj46x\") pod \"e2e-ait-3951eb77-openshift-default-85859ff58c-gf998\" (UID: \"60bdc2d2-b519-4565-8375-61a5254e4f54\") " pod="openshift-ingress/e2e-ait-3951eb77-openshift-default-85859ff58c-gf998"
Nov 01 06:09:20.093294 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:20.093138    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60bdc2d2-b519-4565-8375-61a5254e4f54-istio-podinfo\") pod \"e2e-ait-3951eb77-openshift-default-85859ff58c-gf998\" (UID: \"60bdc2d2-b519-4565-8375-61a5254e4f54\") " pod="openshift-ingress/e2e-ait-3951eb77-openshift-default-85859ff58c-gf998"
Nov 01 06:09:20.093294 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:20.093197    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60bdc2d2-b519-4565-8375-61a5254e4f54-workload-socket\") pod \"e2e-ait-3951eb77-openshift-default-85859ff58c-gf998\" (UID: \"60bdc2d2-b519-4565-8375-61a5254e4f54\") " pod="openshift-ingress/e2e-ait-3951eb77-openshift-default-85859ff58c-gf998"
Nov 01 06:09:20.093294 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:20.093236    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60bdc2d2-b519-4565-8375-61a5254e4f54-workload-certs\") pod \"e2e-ait-3951eb77-openshift-default-85859ff58c-gf998\" (UID: \"60bdc2d2-b519-4565-8375-61a5254e4f54\") " pod="openshift-ingress/e2e-ait-3951eb77-openshift-default-85859ff58c-gf998"
Nov 01 06:09:20.093294 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:20.093264    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60bdc2d2-b519-4565-8375-61a5254e4f54-istio-data\") pod \"e2e-ait-3951eb77-openshift-default-85859ff58c-gf998\" (UID: \"60bdc2d2-b519-4565-8375-61a5254e4f54\") " pod="openshift-ingress/e2e-ait-3951eb77-openshift-default-85859ff58c-gf998"
Nov 01 06:09:20.093531 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:20.093297    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60bdc2d2-b519-4565-8375-61a5254e4f54-credential-socket\") pod \"e2e-ait-3951eb77-openshift-default-85859ff58c-gf998\" (UID: \"60bdc2d2-b519-4565-8375-61a5254e4f54\") " pod="openshift-ingress/e2e-ait-3951eb77-openshift-default-85859ff58c-gf998"
Nov 01 06:09:20.093648 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:20.093622    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60bdc2d2-b519-4565-8375-61a5254e4f54-workload-socket\") pod \"e2e-ait-3951eb77-openshift-default-85859ff58c-gf998\" (UID: \"60bdc2d2-b519-4565-8375-61a5254e4f54\") " pod="openshift-ingress/e2e-ait-3951eb77-openshift-default-85859ff58c-gf998"
Nov 01 06:09:20.093713 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:20.093657    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60bdc2d2-b519-4565-8375-61a5254e4f54-workload-certs\") pod \"e2e-ait-3951eb77-openshift-default-85859ff58c-gf998\" (UID: \"60bdc2d2-b519-4565-8375-61a5254e4f54\") " pod="openshift-ingress/e2e-ait-3951eb77-openshift-default-85859ff58c-gf998"
Nov 01 06:09:20.093713 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:20.093703    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60bdc2d2-b519-4565-8375-61a5254e4f54-istio-data\") pod \"e2e-ait-3951eb77-openshift-default-85859ff58c-gf998\" (UID: \"60bdc2d2-b519-4565-8375-61a5254e4f54\") " pod="openshift-ingress/e2e-ait-3951eb77-openshift-default-85859ff58c-gf998"
Nov 01 06:09:20.093812 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:20.093725    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60bdc2d2-b519-4565-8375-61a5254e4f54-credential-socket\") pod \"e2e-ait-3951eb77-openshift-default-85859ff58c-gf998\" (UID: \"60bdc2d2-b519-4565-8375-61a5254e4f54\") " pod="openshift-ingress/e2e-ait-3951eb77-openshift-default-85859ff58c-gf998"
Nov 01 06:09:20.093867 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:20.093813    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60bdc2d2-b519-4565-8375-61a5254e4f54-istiod-ca-cert\") pod \"e2e-ait-3951eb77-openshift-default-85859ff58c-gf998\" (UID: \"60bdc2d2-b519-4565-8375-61a5254e4f54\") " pod="openshift-ingress/e2e-ait-3951eb77-openshift-default-85859ff58c-gf998"
Nov 01 06:09:20.095421 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:20.095400    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60bdc2d2-b519-4565-8375-61a5254e4f54-istio-envoy\") pod \"e2e-ait-3951eb77-openshift-default-85859ff58c-gf998\" (UID: \"60bdc2d2-b519-4565-8375-61a5254e4f54\") " pod="openshift-ingress/e2e-ait-3951eb77-openshift-default-85859ff58c-gf998"
Nov 01 06:09:20.095758 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:20.095737    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60bdc2d2-b519-4565-8375-61a5254e4f54-istio-podinfo\") pod \"e2e-ait-3951eb77-openshift-default-85859ff58c-gf998\" (UID: \"60bdc2d2-b519-4565-8375-61a5254e4f54\") " pod="openshift-ingress/e2e-ait-3951eb77-openshift-default-85859ff58c-gf998"
Nov 01 06:09:20.153536 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:20.153480    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60bdc2d2-b519-4565-8375-61a5254e4f54-istio-token\") pod \"e2e-ait-3951eb77-openshift-default-85859ff58c-gf998\" (UID: \"60bdc2d2-b519-4565-8375-61a5254e4f54\") " pod="openshift-ingress/e2e-ait-3951eb77-openshift-default-85859ff58c-gf998"
Nov 01 06:09:20.153656 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:20.153564    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj46x\" (UniqueName: \"kubernetes.io/projected/60bdc2d2-b519-4565-8375-61a5254e4f54-kube-api-access-pj46x\") pod \"e2e-ait-3951eb77-openshift-default-85859ff58c-gf998\" (UID: \"60bdc2d2-b519-4565-8375-61a5254e4f54\") " pod="openshift-ingress/e2e-ait-3951eb77-openshift-default-85859ff58c-gf998"
Nov 01 06:09:20.215247 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:20.215220    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3951eb77-openshift-default-85859ff58c-gf998"
Nov 01 06:09:20.263492 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:20.263394    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:09:20.651108 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:20.651081    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3951eb77-openshift-default-85859ff58c-gf998"]
Nov 01 06:09:20.654525 ip-10-0-129-92 kubenswrapper[5689]: W1101 06:09:20.654499    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60bdc2d2_b519_4565_8375_61a5254e4f54.slice/crio-0fc92f6fea1780244eff84288ab0e672d3d8bac633001027c20ecf4950118a20 WatchSource:0}: Error finding container 0fc92f6fea1780244eff84288ab0e672d3d8bac633001027c20ecf4950118a20: Status 404 returned error can't find the container with id 0fc92f6fea1780244eff84288ab0e672d3d8bac633001027c20ecf4950118a20
Nov 01 06:09:20.656637 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:20.656610    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 06:09:20.656703 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:20.656669    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 06:09:20.656703 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:20.656695    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 06:09:20.921954 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:20.921862    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3951eb77-openshift-default-85859ff58c-gf998" event={"ID":"60bdc2d2-b519-4565-8375-61a5254e4f54","Type":"ContainerStarted","Data":"e3ea866f2c43d4a42da2a6fc43d03f715a9d1109a48a7a2523ea7625b1acd0b7"}
Nov 01 06:09:20.921954 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:20.921919    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3951eb77-openshift-default-85859ff58c-gf998" event={"ID":"60bdc2d2-b519-4565-8375-61a5254e4f54","Type":"ContainerStarted","Data":"0fc92f6fea1780244eff84288ab0e672d3d8bac633001027c20ecf4950118a20"}
Nov 01 06:09:20.968834 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:20.968806    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:09:21.042196 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:21.042141    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-3951eb77-openshift-default-85859ff58c-gf998" podStartSLOduration=2.042124399 podStartE2EDuration="2.042124399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:09:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:09:21.040991499 +0000 UTC m=+3939.111788218" watchObservedRunningTime="2026-11-01 06:09:21.042124399 +0000 UTC m=+3939.112921120"
Nov 01 06:09:21.216199 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:21.216122    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3951eb77-openshift-default-85859ff58c-gf998"
Nov 01 06:09:21.221009 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:21.220986    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-3951eb77-openshift-default-85859ff58c-gf998"
Nov 01 06:09:21.925877 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:21.925847    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-3951eb77-openshift-default-85859ff58c-gf998"
Nov 01 06:09:21.926765 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:21.926744    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-3951eb77-openshift-default-85859ff58c-gf998"
Nov 01 06:09:23.109085 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:23.109053    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:09:23.293705 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:23.293673    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:09:23.476196 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:23.476161    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:09:26.403356 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:26.403322    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:09:26.650607 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:26.650569    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:09:26.870719 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:26.870639    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3951eb77-77c58c7955-8n2p7"]
Nov 01 06:09:26.874383 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:26.874364    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3951eb77-77c58c7955-8n2p7"
Nov 01 06:09:26.926893 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:26.926869    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-3951eb77-dockercfg-hgdlg\""
Nov 01 06:09:26.954006 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:26.953968    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72t2l\" (UniqueName: \"kubernetes.io/projected/494650d5-0b43-40d3-b4f0-125df553c510-kube-api-access-72t2l\") pod \"payload-pre-processing-e2e-ait-3951eb77-77c58c7955-8n2p7\" (UID: \"494650d5-0b43-40d3-b4f0-125df553c510\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3951eb77-77c58c7955-8n2p7"
Nov 01 06:09:26.954006 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:26.954002    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/494650d5-0b43-40d3-b4f0-125df553c510-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3951eb77-77c58c7955-8n2p7\" (UID: \"494650d5-0b43-40d3-b4f0-125df553c510\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3951eb77-77c58c7955-8n2p7"
Nov 01 06:09:26.976372 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:26.976346    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-3951eb77\""
Nov 01 06:09:26.977713 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:26.977684    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3951eb77-558dd68b9c-2dzqh"]
Nov 01 06:09:26.982081 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:26.982054    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3951eb77-77c58c7955-8n2p7"]
Nov 01 06:09:26.982225 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:26.982173    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3951eb77-558dd68b9c-2dzqh"
Nov 01 06:09:26.997807 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:26.997788    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3951eb77-558dd68b9c-2dzqh"]
Nov 01 06:09:27.055345 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:27.055316    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-988xs\" (UniqueName: \"kubernetes.io/projected/eb96ab56-077e-471e-b64e-e336783aaa29-kube-api-access-988xs\") pod \"payload-processing-e2e-ait-3951eb77-558dd68b9c-2dzqh\" (UID: \"eb96ab56-077e-471e-b64e-e336783aaa29\") " pod="openshift-ingress/payload-processing-e2e-ait-3951eb77-558dd68b9c-2dzqh"
Nov 01 06:09:27.055519 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:27.055359    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72t2l\" (UniqueName: \"kubernetes.io/projected/494650d5-0b43-40d3-b4f0-125df553c510-kube-api-access-72t2l\") pod \"payload-pre-processing-e2e-ait-3951eb77-77c58c7955-8n2p7\" (UID: \"494650d5-0b43-40d3-b4f0-125df553c510\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3951eb77-77c58c7955-8n2p7"
Nov 01 06:09:27.055519 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:27.055379    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/494650d5-0b43-40d3-b4f0-125df553c510-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3951eb77-77c58c7955-8n2p7\" (UID: \"494650d5-0b43-40d3-b4f0-125df553c510\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3951eb77-77c58c7955-8n2p7"
Nov 01 06:09:27.055638 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:27.055519    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb96ab56-077e-471e-b64e-e336783aaa29-plugins-config-volume\") pod \"payload-processing-e2e-ait-3951eb77-558dd68b9c-2dzqh\" (UID: \"eb96ab56-077e-471e-b64e-e336783aaa29\") " pod="openshift-ingress/payload-processing-e2e-ait-3951eb77-558dd68b9c-2dzqh"
Nov 01 06:09:27.056050 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:27.056031    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/494650d5-0b43-40d3-b4f0-125df553c510-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3951eb77-77c58c7955-8n2p7\" (UID: \"494650d5-0b43-40d3-b4f0-125df553c510\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3951eb77-77c58c7955-8n2p7"
Nov 01 06:09:27.116075 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:27.116047    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72t2l\" (UniqueName: \"kubernetes.io/projected/494650d5-0b43-40d3-b4f0-125df553c510-kube-api-access-72t2l\") pod \"payload-pre-processing-e2e-ait-3951eb77-77c58c7955-8n2p7\" (UID: \"494650d5-0b43-40d3-b4f0-125df553c510\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3951eb77-77c58c7955-8n2p7"
Nov 01 06:09:27.156362 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:27.156332    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb96ab56-077e-471e-b64e-e336783aaa29-plugins-config-volume\") pod \"payload-processing-e2e-ait-3951eb77-558dd68b9c-2dzqh\" (UID: \"eb96ab56-077e-471e-b64e-e336783aaa29\") " pod="openshift-ingress/payload-processing-e2e-ait-3951eb77-558dd68b9c-2dzqh"
Nov 01 06:09:27.156500 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:27.156407    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-988xs\" (UniqueName: \"kubernetes.io/projected/eb96ab56-077e-471e-b64e-e336783aaa29-kube-api-access-988xs\") pod \"payload-processing-e2e-ait-3951eb77-558dd68b9c-2dzqh\" (UID: \"eb96ab56-077e-471e-b64e-e336783aaa29\") " pod="openshift-ingress/payload-processing-e2e-ait-3951eb77-558dd68b9c-2dzqh"
Nov 01 06:09:27.157026 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:27.157008    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb96ab56-077e-471e-b64e-e336783aaa29-plugins-config-volume\") pod \"payload-processing-e2e-ait-3951eb77-558dd68b9c-2dzqh\" (UID: \"eb96ab56-077e-471e-b64e-e336783aaa29\") " pod="openshift-ingress/payload-processing-e2e-ait-3951eb77-558dd68b9c-2dzqh"
Nov 01 06:09:27.187276 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:27.187238    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3951eb77-77c58c7955-8n2p7"
Nov 01 06:09:27.217220 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:27.217196    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-988xs\" (UniqueName: \"kubernetes.io/projected/eb96ab56-077e-471e-b64e-e336783aaa29-kube-api-access-988xs\") pod \"payload-processing-e2e-ait-3951eb77-558dd68b9c-2dzqh\" (UID: \"eb96ab56-077e-471e-b64e-e336783aaa29\") " pod="openshift-ingress/payload-processing-e2e-ait-3951eb77-558dd68b9c-2dzqh"
Nov 01 06:09:27.302485 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:27.302435    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3951eb77-558dd68b9c-2dzqh"
Nov 01 06:09:27.559967 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:27.559900    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3951eb77-77c58c7955-8n2p7"]
Nov 01 06:09:27.563198 ip-10-0-129-92 kubenswrapper[5689]: W1101 06:09:27.563172    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod494650d5_0b43_40d3_b4f0_125df553c510.slice/crio-0f190896d572190f9269f03d95878ba8d38183ed1225f89157a43a4307fb142f WatchSource:0}: Error finding container 0f190896d572190f9269f03d95878ba8d38183ed1225f89157a43a4307fb142f: Status 404 returned error can't find the container with id 0f190896d572190f9269f03d95878ba8d38183ed1225f89157a43a4307fb142f
Nov 01 06:09:27.674050 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:27.674026    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3951eb77-558dd68b9c-2dzqh"]
Nov 01 06:09:27.678314 ip-10-0-129-92 kubenswrapper[5689]: W1101 06:09:27.678289    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb96ab56_077e_471e_b64e_e336783aaa29.slice/crio-6c7fbb943d1e21a653b9e43a9997c01e2bef2a2f7cf16167666466b3a2461957 WatchSource:0}: Error finding container 6c7fbb943d1e21a653b9e43a9997c01e2bef2a2f7cf16167666466b3a2461957: Status 404 returned error can't find the container with id 6c7fbb943d1e21a653b9e43a9997c01e2bef2a2f7cf16167666466b3a2461957
Nov 01 06:09:27.953553 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:27.953517    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3951eb77-77c58c7955-8n2p7" event={"ID":"494650d5-0b43-40d3-b4f0-125df553c510","Type":"ContainerStarted","Data":"d991d65bd1fb41620dfdbee321d8ec982853f5e67f5777f9abdefcb41d0111d4"}
Nov 01 06:09:27.953728 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:27.953581    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3951eb77-77c58c7955-8n2p7" event={"ID":"494650d5-0b43-40d3-b4f0-125df553c510","Type":"ContainerStarted","Data":"0f190896d572190f9269f03d95878ba8d38183ed1225f89157a43a4307fb142f"}
Nov 01 06:09:27.953728 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:27.953609    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-3951eb77-77c58c7955-8n2p7"
Nov 01 06:09:27.955097 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:27.955074    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3951eb77-558dd68b9c-2dzqh" event={"ID":"eb96ab56-077e-471e-b64e-e336783aaa29","Type":"ContainerStarted","Data":"aba6f033923cec58024223121d45ca40c460276210c663f95355c757e2cf6b17"}
Nov 01 06:09:27.955347 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:27.955111    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3951eb77-558dd68b9c-2dzqh" event={"ID":"eb96ab56-077e-471e-b64e-e336783aaa29","Type":"ContainerStarted","Data":"6c7fbb943d1e21a653b9e43a9997c01e2bef2a2f7cf16167666466b3a2461957"}
Nov 01 06:09:27.955347 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:27.955201    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-3951eb77-558dd68b9c-2dzqh"
Nov 01 06:09:28.069030 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:28.068981    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-3951eb77-77c58c7955-8n2p7" podStartSLOduration=2.068967585 podStartE2EDuration="2.068967585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:09:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:09:28.06830065 +0000 UTC m=+3946.139097393" watchObservedRunningTime="2026-11-01 06:09:28.068967585 +0000 UTC m=+3946.139764302"
Nov 01 06:09:29.323194 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:29.323127    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-3951eb77-558dd68b9c-2dzqh" podStartSLOduration=3.323109276 podStartE2EDuration="3.323109276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:09:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:09:28.188175269 +0000 UTC m=+3946.258972005" watchObservedRunningTime="2026-11-01 06:09:29.323109276 +0000 UTC m=+3947.393905997"
Nov 01 06:09:29.323683 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:29.323664    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3951eb77-5cd7d8bbf8-c6f8w"]
Nov 01 06:09:29.327179 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:29.327162    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3951eb77-5cd7d8bbf8-c6f8w"
Nov 01 06:09:29.375887 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:29.375858    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9705308-9add-4b52-86fb-6d5e4b3206c1-metrics-certs\") pod \"maas-api-e2e-ait-3951eb77-5cd7d8bbf8-c6f8w\" (UID: \"c9705308-9add-4b52-86fb-6d5e4b3206c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3951eb77-5cd7d8bbf8-c6f8w"
Nov 01 06:09:29.376001 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:29.375974    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9705308-9add-4b52-86fb-6d5e4b3206c1-maas-api-tls\") pod \"maas-api-e2e-ait-3951eb77-5cd7d8bbf8-c6f8w\" (UID: \"c9705308-9add-4b52-86fb-6d5e4b3206c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3951eb77-5cd7d8bbf8-c6f8w"
Nov 01 06:09:29.376043 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:29.376003    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsgxb\" (UniqueName: \"kubernetes.io/projected/c9705308-9add-4b52-86fb-6d5e4b3206c1-kube-api-access-bsgxb\") pod \"maas-api-e2e-ait-3951eb77-5cd7d8bbf8-c6f8w\" (UID: \"c9705308-9add-4b52-86fb-6d5e4b3206c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3951eb77-5cd7d8bbf8-c6f8w"
Nov 01 06:09:29.376043 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:29.376030    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9705308-9add-4b52-86fb-6d5e4b3206c1-openshift-service-ca\") pod \"maas-api-e2e-ait-3951eb77-5cd7d8bbf8-c6f8w\" (UID: \"c9705308-9add-4b52-86fb-6d5e4b3206c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3951eb77-5cd7d8bbf8-c6f8w"
Nov 01 06:09:29.428429 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:29.428405    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-3951eb77-serving-cert\""
Nov 01 06:09:29.428636 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:29.428531    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9729w\""
Nov 01 06:09:29.428636 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:29.428613    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 01 06:09:29.430341 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:29.430320    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3951eb77-5cd7d8bbf8-c6f8w"]
Nov 01 06:09:29.477125 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:29.477099    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9705308-9add-4b52-86fb-6d5e4b3206c1-maas-api-tls\") pod \"maas-api-e2e-ait-3951eb77-5cd7d8bbf8-c6f8w\" (UID: \"c9705308-9add-4b52-86fb-6d5e4b3206c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3951eb77-5cd7d8bbf8-c6f8w"
Nov 01 06:09:29.477125 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:29.477130    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsgxb\" (UniqueName: \"kubernetes.io/projected/c9705308-9add-4b52-86fb-6d5e4b3206c1-kube-api-access-bsgxb\") pod \"maas-api-e2e-ait-3951eb77-5cd7d8bbf8-c6f8w\" (UID: \"c9705308-9add-4b52-86fb-6d5e4b3206c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3951eb77-5cd7d8bbf8-c6f8w"
Nov 01 06:09:29.477363 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:29.477151    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9705308-9add-4b52-86fb-6d5e4b3206c1-openshift-service-ca\") pod \"maas-api-e2e-ait-3951eb77-5cd7d8bbf8-c6f8w\" (UID: \"c9705308-9add-4b52-86fb-6d5e4b3206c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3951eb77-5cd7d8bbf8-c6f8w"
Nov 01 06:09:29.477363 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:29.477212    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9705308-9add-4b52-86fb-6d5e4b3206c1-metrics-certs\") pod \"maas-api-e2e-ait-3951eb77-5cd7d8bbf8-c6f8w\" (UID: \"c9705308-9add-4b52-86fb-6d5e4b3206c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3951eb77-5cd7d8bbf8-c6f8w"
Nov 01 06:09:29.477363 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:09:29.477248    5689 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-3951eb77-serving-cert: secret "maas-api-e2e-ait-3951eb77-serving-cert" not found
Nov 01 06:09:29.477363 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:09:29.477317    5689 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9705308-9add-4b52-86fb-6d5e4b3206c1-maas-api-tls podName:c9705308-9add-4b52-86fb-6d5e4b3206c1 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:29.977297156 +0000 UTC m=+3948.048093854 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c9705308-9add-4b52-86fb-6d5e4b3206c1-maas-api-tls") pod "maas-api-e2e-ait-3951eb77-5cd7d8bbf8-c6f8w" (UID: "c9705308-9add-4b52-86fb-6d5e4b3206c1") : secret "maas-api-e2e-ait-3951eb77-serving-cert" not found
Nov 01 06:09:29.478120 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:29.478020    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9705308-9add-4b52-86fb-6d5e4b3206c1-openshift-service-ca\") pod \"maas-api-e2e-ait-3951eb77-5cd7d8bbf8-c6f8w\" (UID: \"c9705308-9add-4b52-86fb-6d5e4b3206c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3951eb77-5cd7d8bbf8-c6f8w"
Nov 01 06:09:29.479833 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:29.479805    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9705308-9add-4b52-86fb-6d5e4b3206c1-metrics-certs\") pod \"maas-api-e2e-ait-3951eb77-5cd7d8bbf8-c6f8w\" (UID: \"c9705308-9add-4b52-86fb-6d5e4b3206c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3951eb77-5cd7d8bbf8-c6f8w"
Nov 01 06:09:29.537937 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:29.537910    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsgxb\" (UniqueName: \"kubernetes.io/projected/c9705308-9add-4b52-86fb-6d5e4b3206c1-kube-api-access-bsgxb\") pod \"maas-api-e2e-ait-3951eb77-5cd7d8bbf8-c6f8w\" (UID: \"c9705308-9add-4b52-86fb-6d5e4b3206c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3951eb77-5cd7d8bbf8-c6f8w"
Nov 01 06:09:29.555404 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:29.555377    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:09:29.739976 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:29.739939    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:09:29.916211 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:29.916175    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:09:29.982143 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:29.982113    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9705308-9add-4b52-86fb-6d5e4b3206c1-maas-api-tls\") pod \"maas-api-e2e-ait-3951eb77-5cd7d8bbf8-c6f8w\" (UID: \"c9705308-9add-4b52-86fb-6d5e4b3206c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3951eb77-5cd7d8bbf8-c6f8w"
Nov 01 06:09:29.984653 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:29.984631    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9705308-9add-4b52-86fb-6d5e4b3206c1-maas-api-tls\") pod \"maas-api-e2e-ait-3951eb77-5cd7d8bbf8-c6f8w\" (UID: \"c9705308-9add-4b52-86fb-6d5e4b3206c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3951eb77-5cd7d8bbf8-c6f8w"
Nov 01 06:09:30.068365 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:09:30.068283    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 06:09:30.241478 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:30.241428    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3951eb77-5cd7d8bbf8-c6f8w"
Nov 01 06:09:30.618283 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:30.618255    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3951eb77-5cd7d8bbf8-c6f8w"]
Nov 01 06:09:30.621941 ip-10-0-129-92 kubenswrapper[5689]: W1101 06:09:30.621911    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9705308_9add_4b52_86fb_6d5e4b3206c1.slice/crio-42323ff6007418eab2e8e6ecab300edd55222c1abde994082acf120a8d1e1c31 WatchSource:0}: Error finding container 42323ff6007418eab2e8e6ecab300edd55222c1abde994082acf120a8d1e1c31: Status 404 returned error can't find the container with id 42323ff6007418eab2e8e6ecab300edd55222c1abde994082acf120a8d1e1c31
Nov 01 06:09:30.623904 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:30.623885    5689 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 06:09:30.969024 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:30.968983    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3951eb77-5cd7d8bbf8-c6f8w" event={"ID":"c9705308-9add-4b52-86fb-6d5e4b3206c1","Type":"ContainerStarted","Data":"42323ff6007418eab2e8e6ecab300edd55222c1abde994082acf120a8d1e1c31"}
Nov 01 06:09:31.974406 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:31.974366    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3951eb77-5cd7d8bbf8-c6f8w" event={"ID":"c9705308-9add-4b52-86fb-6d5e4b3206c1","Type":"ContainerStarted","Data":"27cb9efb6022691cd48a40efcf561b20230b46638985145360a3f5b68c23004e"}
Nov 01 06:09:31.974947 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:31.974495    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3951eb77-5cd7d8bbf8-c6f8w"
Nov 01 06:09:32.092508 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:32.092427    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3951eb77-5cd7d8bbf8-c6f8w" podStartSLOduration=2.210397591 podStartE2EDuration="3.092409203s" totalImagesPullingTime="882.011612ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:09:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:09:32.090669379 +0000 UTC m=+3950.161466100" watchObservedRunningTime="2026-11-01 06:09:32.092409203 +0000 UTC m=+3950.163205924"
Nov 01 06:09:34.480380 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:34.480348    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:09:37.472573 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:37.472539    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:09:37.983944 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:37.983917    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3951eb77-5cd7d8bbf8-c6f8w"
Nov 01 06:09:38.962245 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:38.962215    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-3951eb77-77c58c7955-8n2p7"
Nov 01 06:09:38.962668 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:38.962279    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-3951eb77-558dd68b9c-2dzqh"
Nov 01 06:09:39.304416 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:39.304329    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:09:44.086979 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:44.086951    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:09:48.046381 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:48.046349    5689 generic.go:356] "Generic (PLEG): container finished" podID="60bdc2d2-b519-4565-8375-61a5254e4f54" containerID="e3ea866f2c43d4a42da2a6fc43d03f715a9d1109a48a7a2523ea7625b1acd0b7" exitCode=0
Nov 01 06:09:48.046830 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:48.046422    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3951eb77-openshift-default-85859ff58c-gf998" event={"ID":"60bdc2d2-b519-4565-8375-61a5254e4f54","Type":"ContainerDied","Data":"e3ea866f2c43d4a42da2a6fc43d03f715a9d1109a48a7a2523ea7625b1acd0b7"}
Nov 01 06:09:48.046830 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:48.046774    5689 scope.go:122] "RemoveContainer" containerID="e3ea866f2c43d4a42da2a6fc43d03f715a9d1109a48a7a2523ea7625b1acd0b7"
Nov 01 06:09:48.047909 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:48.047884    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 06:09:48.047976 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:48.047949    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 06:09:48.048020 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:48.047980    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 06:09:49.061445 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:49.061408    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3951eb77-openshift-default-85859ff58c-gf998" event={"ID":"60bdc2d2-b519-4565-8375-61a5254e4f54","Type":"ContainerStarted","Data":"f92e65f201136a6cb9dabf193975702518c2b6f9ac3cbb395125e99e329e17e9"}
Nov 01 06:09:49.151750 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:49.151723    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:09:49.215867 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:49.215827    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3951eb77-openshift-default-85859ff58c-gf998"
Nov 01 06:09:49.216883 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:49.216856    5689 patch_prober.go:29] interesting pod/e2e-ait-3951eb77-openshift-default-85859ff58c-gf998 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body=
Nov 01 06:09:49.217004 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:49.216903    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3951eb77-openshift-default-85859ff58c-gf998" podUID="60bdc2d2-b519-4565-8375-61a5254e4f54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused"
Nov 01 06:09:50.216649 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:50.216565    5689 patch_prober.go:29] interesting pod/e2e-ait-3951eb77-openshift-default-85859ff58c-gf998 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body=
Nov 01 06:09:50.216649 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:50.216626    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3951eb77-openshift-default-85859ff58c-gf998" podUID="60bdc2d2-b519-4565-8375-61a5254e4f54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused"
Nov 01 06:09:51.215775 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:51.215738    5689 patch_prober.go:29] interesting pod/e2e-ait-3951eb77-openshift-default-85859ff58c-gf998 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body=
Nov 01 06:09:51.215967 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:51.215793    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3951eb77-openshift-default-85859ff58c-gf998" podUID="60bdc2d2-b519-4565-8375-61a5254e4f54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused"
Nov 01 06:09:51.926383 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:51.926344    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-3951eb77-openshift-default-85859ff58c-gf998"
Nov 01 06:09:52.215956 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:52.215877    5689 patch_prober.go:29] interesting pod/e2e-ait-3951eb77-openshift-default-85859ff58c-gf998 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body=
Nov 01 06:09:52.215956 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:52.215934    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3951eb77-openshift-default-85859ff58c-gf998" podUID="60bdc2d2-b519-4565-8375-61a5254e4f54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused"
Nov 01 06:09:53.215934 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:53.215899    5689 patch_prober.go:29] interesting pod/e2e-ait-3951eb77-openshift-default-85859ff58c-gf998 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body=
Nov 01 06:09:53.216344 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:53.215949    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3951eb77-openshift-default-85859ff58c-gf998" podUID="60bdc2d2-b519-4565-8375-61a5254e4f54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused"
Nov 01 06:09:54.126177 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:54.126113    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:09:54.220093 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:54.220065    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-3951eb77-openshift-default-85859ff58c-gf998"
Nov 01 06:09:54.221080 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:54.221059    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-3951eb77-openshift-default-85859ff58c-gf998"
Nov 01 06:09:58.912276 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:09:58.912240    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:10:00.068730 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:10:00.068693    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 06:10:03.639130 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:03.639053    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:10:08.404447 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:08.404414    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:10:10.774951 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:10.774915    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:10:11.059687 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:11.059589    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3951eb77-558dd68b9c-2dzqh"]
Nov 01 06:10:11.169798 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:11.169766    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3951eb77-77c58c7955-8n2p7"]
Nov 01 06:10:11.461882 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:11.461849    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:10:11.645713 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:11.645675    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:10:13.126211 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:13.126173    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:10:16.059916 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:16.059877    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-3951eb77-558dd68b9c-2dzqh" podUID="eb96ab56-077e-471e-b64e-e336783aaa29" containerName="payload-processing" containerID="cri-o://aba6f033923cec58024223121d45ca40c460276210c663f95355c757e2cf6b17" gracePeriod=25
Nov 01 06:10:16.170745 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:16.170684    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-3951eb77-77c58c7955-8n2p7" podUID="494650d5-0b43-40d3-b4f0-125df553c510" containerName="payload-pre-processing" containerID="cri-o://d991d65bd1fb41620dfdbee321d8ec982853f5e67f5777f9abdefcb41d0111d4" gracePeriod=25
Nov 01 06:10:16.204847 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:16.204810    5689 generic.go:356] "Generic (PLEG): container finished" podID="eb96ab56-077e-471e-b64e-e336783aaa29" containerID="aba6f033923cec58024223121d45ca40c460276210c663f95355c757e2cf6b17" exitCode=0
Nov 01 06:10:16.204981 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:16.204883    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3951eb77-558dd68b9c-2dzqh" event={"ID":"eb96ab56-077e-471e-b64e-e336783aaa29","Type":"ContainerDied","Data":"aba6f033923cec58024223121d45ca40c460276210c663f95355c757e2cf6b17"}
Nov 01 06:10:16.308943 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:16.308922    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3951eb77-558dd68b9c-2dzqh"
Nov 01 06:10:16.410811 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:16.410787    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3951eb77-77c58c7955-8n2p7"
Nov 01 06:10:16.436770 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:16.436747    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb96ab56-077e-471e-b64e-e336783aaa29-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb96ab56-077e-471e-b64e-e336783aaa29-plugins-config-volume\") pod \"eb96ab56-077e-471e-b64e-e336783aaa29\" (UID: \"eb96ab56-077e-471e-b64e-e336783aaa29\") "
Nov 01 06:10:16.436919 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:16.436789    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb96ab56-077e-471e-b64e-e336783aaa29-kube-api-access-988xs\" (UniqueName: \"kubernetes.io/projected/eb96ab56-077e-471e-b64e-e336783aaa29-kube-api-access-988xs\") pod \"eb96ab56-077e-471e-b64e-e336783aaa29\" (UID: \"eb96ab56-077e-471e-b64e-e336783aaa29\") "
Nov 01 06:10:16.437193 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:16.437166    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb96ab56-077e-471e-b64e-e336783aaa29-plugins-config-volume" pod "eb96ab56-077e-471e-b64e-e336783aaa29" (UID: "eb96ab56-077e-471e-b64e-e336783aaa29"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:10:16.439063 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:16.439038    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb96ab56-077e-471e-b64e-e336783aaa29-kube-api-access-988xs" pod "eb96ab56-077e-471e-b64e-e336783aaa29" (UID: "eb96ab56-077e-471e-b64e-e336783aaa29"). InnerVolumeSpecName "kube-api-access-988xs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:10:16.465720 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:16.465696    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:10:16.538202 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:16.538167    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/494650d5-0b43-40d3-b4f0-125df553c510-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/494650d5-0b43-40d3-b4f0-125df553c510-plugins-config-volume\") pod \"494650d5-0b43-40d3-b4f0-125df553c510\" (UID: \"494650d5-0b43-40d3-b4f0-125df553c510\") "
Nov 01 06:10:16.538384 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:16.538229    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/494650d5-0b43-40d3-b4f0-125df553c510-kube-api-access-72t2l\" (UniqueName: \"kubernetes.io/projected/494650d5-0b43-40d3-b4f0-125df553c510-kube-api-access-72t2l\") pod \"494650d5-0b43-40d3-b4f0-125df553c510\" (UID: \"494650d5-0b43-40d3-b4f0-125df553c510\") "
Nov 01 06:10:16.538599 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:16.538572    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/494650d5-0b43-40d3-b4f0-125df553c510-plugins-config-volume" pod "494650d5-0b43-40d3-b4f0-125df553c510" (UID: "494650d5-0b43-40d3-b4f0-125df553c510"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:10:16.538768 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:16.538732    5689 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb96ab56-077e-471e-b64e-e336783aaa29-plugins-config-volume\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:10:16.538917 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:16.538781    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-988xs\" (UniqueName: \"kubernetes.io/projected/eb96ab56-077e-471e-b64e-e336783aaa29-kube-api-access-988xs\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:10:16.538917 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:16.538797    5689 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/494650d5-0b43-40d3-b4f0-125df553c510-plugins-config-volume\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:10:16.540912 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:16.540884    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/494650d5-0b43-40d3-b4f0-125df553c510-kube-api-access-72t2l" pod "494650d5-0b43-40d3-b4f0-125df553c510" (UID: "494650d5-0b43-40d3-b4f0-125df553c510"). InnerVolumeSpecName "kube-api-access-72t2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:10:16.639786 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:16.639712    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72t2l\" (UniqueName: \"kubernetes.io/projected/494650d5-0b43-40d3-b4f0-125df553c510-kube-api-access-72t2l\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:10:17.170823 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:17.170790    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3951eb77-5cd7d8bbf8-c6f8w"]
Nov 01 06:10:17.171204 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:17.170980    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3951eb77-5cd7d8bbf8-c6f8w" podUID="c9705308-9add-4b52-86fb-6d5e4b3206c1" containerName="maas-api" containerID="cri-o://27cb9efb6022691cd48a40efcf561b20230b46638985145360a3f5b68c23004e" gracePeriod=30
Nov 01 06:10:17.211163 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:17.210909    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3951eb77-558dd68b9c-2dzqh"
Nov 01 06:10:17.211163 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:17.210997    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3951eb77-558dd68b9c-2dzqh" event={"ID":"eb96ab56-077e-471e-b64e-e336783aaa29","Type":"ContainerDied","Data":"6c7fbb943d1e21a653b9e43a9997c01e2bef2a2f7cf16167666466b3a2461957"}
Nov 01 06:10:17.211163 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:17.211057    5689 scope.go:122] "RemoveContainer" containerID="aba6f033923cec58024223121d45ca40c460276210c663f95355c757e2cf6b17"
Nov 01 06:10:17.212628 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:17.212605    5689 generic.go:356] "Generic (PLEG): container finished" podID="494650d5-0b43-40d3-b4f0-125df553c510" containerID="d991d65bd1fb41620dfdbee321d8ec982853f5e67f5777f9abdefcb41d0111d4" exitCode=0
Nov 01 06:10:17.212734 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:17.212650    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3951eb77-77c58c7955-8n2p7"
Nov 01 06:10:17.212734 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:17.212674    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3951eb77-77c58c7955-8n2p7" event={"ID":"494650d5-0b43-40d3-b4f0-125df553c510","Type":"ContainerDied","Data":"d991d65bd1fb41620dfdbee321d8ec982853f5e67f5777f9abdefcb41d0111d4"}
Nov 01 06:10:17.212734 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:17.212714    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3951eb77-77c58c7955-8n2p7" event={"ID":"494650d5-0b43-40d3-b4f0-125df553c510","Type":"ContainerDied","Data":"0f190896d572190f9269f03d95878ba8d38183ed1225f89157a43a4307fb142f"}
Nov 01 06:10:17.282381 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:17.282361    5689 scope.go:122] "RemoveContainer" containerID="d991d65bd1fb41620dfdbee321d8ec982853f5e67f5777f9abdefcb41d0111d4"
Nov 01 06:10:17.291979 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:17.291956    5689 scope.go:122] "RemoveContainer" containerID="d991d65bd1fb41620dfdbee321d8ec982853f5e67f5777f9abdefcb41d0111d4"
Nov 01 06:10:17.292275 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:10:17.292254    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d991d65bd1fb41620dfdbee321d8ec982853f5e67f5777f9abdefcb41d0111d4\": container with ID starting with d991d65bd1fb41620dfdbee321d8ec982853f5e67f5777f9abdefcb41d0111d4 not found: ID does not exist" containerID="d991d65bd1fb41620dfdbee321d8ec982853f5e67f5777f9abdefcb41d0111d4"
Nov 01 06:10:17.292341 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:17.292285    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d991d65bd1fb41620dfdbee321d8ec982853f5e67f5777f9abdefcb41d0111d4"} err="failed to get container status \"d991d65bd1fb41620dfdbee321d8ec982853f5e67f5777f9abdefcb41d0111d4\": rpc error: code = NotFound desc = could not find container \"d991d65bd1fb41620dfdbee321d8ec982853f5e67f5777f9abdefcb41d0111d4\": container with ID starting with d991d65bd1fb41620dfdbee321d8ec982853f5e67f5777f9abdefcb41d0111d4 not found: ID does not exist"
Nov 01 06:10:17.308960 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:17.308933    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:10:17.397170 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:17.397138    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3951eb77-77c58c7955-8n2p7"]
Nov 01 06:10:17.402712 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:17.402689    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3951eb77-77c58c7955-8n2p7"]
Nov 01 06:10:17.420189 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:17.420170    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3951eb77-5cd7d8bbf8-c6f8w"
Nov 01 06:10:17.511408 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:17.511379    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3951eb77-558dd68b9c-2dzqh"]
Nov 01 06:10:17.514962 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:17.514937    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3951eb77-558dd68b9c-2dzqh"]
Nov 01 06:10:17.548056 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:17.548030    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9705308-9add-4b52-86fb-6d5e4b3206c1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9705308-9add-4b52-86fb-6d5e4b3206c1-openshift-service-ca\") pod \"c9705308-9add-4b52-86fb-6d5e4b3206c1\" (UID: \"c9705308-9add-4b52-86fb-6d5e4b3206c1\") "
Nov 01 06:10:17.548201 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:17.548065    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9705308-9add-4b52-86fb-6d5e4b3206c1-kube-api-access-bsgxb\" (UniqueName: \"kubernetes.io/projected/c9705308-9add-4b52-86fb-6d5e4b3206c1-kube-api-access-bsgxb\") pod \"c9705308-9add-4b52-86fb-6d5e4b3206c1\" (UID: \"c9705308-9add-4b52-86fb-6d5e4b3206c1\") "
Nov 01 06:10:17.548201 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:17.548086    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9705308-9add-4b52-86fb-6d5e4b3206c1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9705308-9add-4b52-86fb-6d5e4b3206c1-metrics-certs\") pod \"c9705308-9add-4b52-86fb-6d5e4b3206c1\" (UID: \"c9705308-9add-4b52-86fb-6d5e4b3206c1\") "
Nov 01 06:10:17.548201 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:17.548134    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9705308-9add-4b52-86fb-6d5e4b3206c1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9705308-9add-4b52-86fb-6d5e4b3206c1-maas-api-tls\") pod \"c9705308-9add-4b52-86fb-6d5e4b3206c1\" (UID: \"c9705308-9add-4b52-86fb-6d5e4b3206c1\") "
Nov 01 06:10:17.548476 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:17.548428    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9705308-9add-4b52-86fb-6d5e4b3206c1-openshift-service-ca" pod "c9705308-9add-4b52-86fb-6d5e4b3206c1" (UID: "c9705308-9add-4b52-86fb-6d5e4b3206c1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:10:17.550970 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:17.550923    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9705308-9add-4b52-86fb-6d5e4b3206c1-maas-api-tls" pod "c9705308-9add-4b52-86fb-6d5e4b3206c1" (UID: "c9705308-9add-4b52-86fb-6d5e4b3206c1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:10:17.551109 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:17.551063    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9705308-9add-4b52-86fb-6d5e4b3206c1-metrics-certs" pod "c9705308-9add-4b52-86fb-6d5e4b3206c1" (UID: "c9705308-9add-4b52-86fb-6d5e4b3206c1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:10:17.551109 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:17.551071    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9705308-9add-4b52-86fb-6d5e4b3206c1-kube-api-access-bsgxb" pod "c9705308-9add-4b52-86fb-6d5e4b3206c1" (UID: "c9705308-9add-4b52-86fb-6d5e4b3206c1"). InnerVolumeSpecName "kube-api-access-bsgxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:10:17.649159 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:17.649128    5689 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9705308-9add-4b52-86fb-6d5e4b3206c1-openshift-service-ca\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:10:17.649159 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:17.649151    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsgxb\" (UniqueName: \"kubernetes.io/projected/c9705308-9add-4b52-86fb-6d5e4b3206c1-kube-api-access-bsgxb\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:10:17.649159 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:17.649162    5689 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9705308-9add-4b52-86fb-6d5e4b3206c1-metrics-certs\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:10:17.649370 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:17.649171    5689 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9705308-9add-4b52-86fb-6d5e4b3206c1-maas-api-tls\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:10:18.066110 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:18.066033    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="494650d5-0b43-40d3-b4f0-125df553c510" path="/var/lib/kubelet/pods/494650d5-0b43-40d3-b4f0-125df553c510/volumes"
Nov 01 06:10:18.066405 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:18.066388    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb96ab56-077e-471e-b64e-e336783aaa29" path="/var/lib/kubelet/pods/eb96ab56-077e-471e-b64e-e336783aaa29/volumes"
Nov 01 06:10:18.220216 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:18.220187    5689 generic.go:356] "Generic (PLEG): container finished" podID="c9705308-9add-4b52-86fb-6d5e4b3206c1" containerID="27cb9efb6022691cd48a40efcf561b20230b46638985145360a3f5b68c23004e" exitCode=0
Nov 01 06:10:18.220672 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:18.220230    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3951eb77-5cd7d8bbf8-c6f8w" event={"ID":"c9705308-9add-4b52-86fb-6d5e4b3206c1","Type":"ContainerDied","Data":"27cb9efb6022691cd48a40efcf561b20230b46638985145360a3f5b68c23004e"}
Nov 01 06:10:18.220672 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:18.220235    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3951eb77-5cd7d8bbf8-c6f8w"
Nov 01 06:10:18.220672 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:18.220274    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3951eb77-5cd7d8bbf8-c6f8w" event={"ID":"c9705308-9add-4b52-86fb-6d5e4b3206c1","Type":"ContainerDied","Data":"42323ff6007418eab2e8e6ecab300edd55222c1abde994082acf120a8d1e1c31"}
Nov 01 06:10:18.220672 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:18.220296    5689 scope.go:122] "RemoveContainer" containerID="27cb9efb6022691cd48a40efcf561b20230b46638985145360a3f5b68c23004e"
Nov 01 06:10:18.228765 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:18.228748    5689 scope.go:122] "RemoveContainer" containerID="27cb9efb6022691cd48a40efcf561b20230b46638985145360a3f5b68c23004e"
Nov 01 06:10:18.229042 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:10:18.229023    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27cb9efb6022691cd48a40efcf561b20230b46638985145360a3f5b68c23004e\": container with ID starting with 27cb9efb6022691cd48a40efcf561b20230b46638985145360a3f5b68c23004e not found: ID does not exist" containerID="27cb9efb6022691cd48a40efcf561b20230b46638985145360a3f5b68c23004e"
Nov 01 06:10:18.229110 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:18.229055    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27cb9efb6022691cd48a40efcf561b20230b46638985145360a3f5b68c23004e"} err="failed to get container status \"27cb9efb6022691cd48a40efcf561b20230b46638985145360a3f5b68c23004e\": rpc error: code = NotFound desc = could not find container \"27cb9efb6022691cd48a40efcf561b20230b46638985145360a3f5b68c23004e\": container with ID starting with 27cb9efb6022691cd48a40efcf561b20230b46638985145360a3f5b68c23004e not found: ID does not exist"
Nov 01 06:10:18.382931 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:18.382861    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3951eb77-5cd7d8bbf8-c6f8w"]
Nov 01 06:10:18.432125 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:18.432096    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3951eb77-5cd7d8bbf8-c6f8w"]
Nov 01 06:10:18.796772 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:18.796732    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:10:19.077252 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:19.077171    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:10:20.066646 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:20.066604    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9705308-9add-4b52-86fb-6d5e4b3206c1" path="/var/lib/kubelet/pods/c9705308-9add-4b52-86fb-6d5e4b3206c1/volumes"
Nov 01 06:10:20.667083 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:20.667054    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3951eb77-openshift-default-85859ff58c-gf998"]
Nov 01 06:10:20.667245 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:20.667231    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-3951eb77-openshift-default-85859ff58c-gf998" podUID="60bdc2d2-b519-4565-8375-61a5254e4f54" containerName="istio-proxy" containerID="cri-o://f92e65f201136a6cb9dabf193975702518c2b6f9ac3cbb395125e99e329e17e9" gracePeriod=30
Nov 01 06:10:20.699073 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:20.699047    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:10:20.928744 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:20.928665    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:10:23.609218 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:23.609186    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:10:24.222412 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:24.222379    5689 patch_prober.go:29] interesting pod/e2e-ait-3951eb77-openshift-default-85859ff58c-gf998 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 06:10:24.222616 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:24.222436    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-3951eb77-openshift-default-85859ff58c-gf998" podUID="60bdc2d2-b519-4565-8375-61a5254e4f54" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 06:10:25.916744 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:25.916715    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3951eb77-openshift-default-85859ff58c-gf998"
Nov 01 06:10:26.019371 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:26.019313    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60bdc2d2-b519-4565-8375-61a5254e4f54-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60bdc2d2-b519-4565-8375-61a5254e4f54-credential-socket\") pod \"60bdc2d2-b519-4565-8375-61a5254e4f54\" (UID: \"60bdc2d2-b519-4565-8375-61a5254e4f54\") "
Nov 01 06:10:26.019371 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:26.019345    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60bdc2d2-b519-4565-8375-61a5254e4f54-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60bdc2d2-b519-4565-8375-61a5254e4f54-istiod-ca-cert\") pod \"60bdc2d2-b519-4565-8375-61a5254e4f54\" (UID: \"60bdc2d2-b519-4565-8375-61a5254e4f54\") "
Nov 01 06:10:26.019608 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:26.019387    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60bdc2d2-b519-4565-8375-61a5254e4f54-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60bdc2d2-b519-4565-8375-61a5254e4f54-istio-envoy\") pod \"60bdc2d2-b519-4565-8375-61a5254e4f54\" (UID: \"60bdc2d2-b519-4565-8375-61a5254e4f54\") "
Nov 01 06:10:26.019608 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:26.019416    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60bdc2d2-b519-4565-8375-61a5254e4f54-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60bdc2d2-b519-4565-8375-61a5254e4f54-workload-certs\") pod \"60bdc2d2-b519-4565-8375-61a5254e4f54\" (UID: \"60bdc2d2-b519-4565-8375-61a5254e4f54\") "
Nov 01 06:10:26.019608 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:26.019494    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/60bdc2d2-b519-4565-8375-61a5254e4f54-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60bdc2d2-b519-4565-8375-61a5254e4f54-istio-podinfo\") pod \"60bdc2d2-b519-4565-8375-61a5254e4f54\" (UID: \"60bdc2d2-b519-4565-8375-61a5254e4f54\") "
Nov 01 06:10:26.019608 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:26.019520    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60bdc2d2-b519-4565-8375-61a5254e4f54-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60bdc2d2-b519-4565-8375-61a5254e4f54-workload-socket\") pod \"60bdc2d2-b519-4565-8375-61a5254e4f54\" (UID: \"60bdc2d2-b519-4565-8375-61a5254e4f54\") "
Nov 01 06:10:26.019608 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:26.019560    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60bdc2d2-b519-4565-8375-61a5254e4f54-istio-token\" (UniqueName: \"kubernetes.io/projected/60bdc2d2-b519-4565-8375-61a5254e4f54-istio-token\") pod \"60bdc2d2-b519-4565-8375-61a5254e4f54\" (UID: \"60bdc2d2-b519-4565-8375-61a5254e4f54\") "
Nov 01 06:10:26.019608 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:26.019606    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60bdc2d2-b519-4565-8375-61a5254e4f54-kube-api-access-pj46x\" (UniqueName: \"kubernetes.io/projected/60bdc2d2-b519-4565-8375-61a5254e4f54-kube-api-access-pj46x\") pod \"60bdc2d2-b519-4565-8375-61a5254e4f54\" (UID: \"60bdc2d2-b519-4565-8375-61a5254e4f54\") "
Nov 01 06:10:26.019922 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:26.019639    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60bdc2d2-b519-4565-8375-61a5254e4f54-credential-socket" pod "60bdc2d2-b519-4565-8375-61a5254e4f54" (UID: "60bdc2d2-b519-4565-8375-61a5254e4f54"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:10:26.019922 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:26.019713    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60bdc2d2-b519-4565-8375-61a5254e4f54-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60bdc2d2-b519-4565-8375-61a5254e4f54-istio-data\") pod \"60bdc2d2-b519-4565-8375-61a5254e4f54\" (UID: \"60bdc2d2-b519-4565-8375-61a5254e4f54\") "
Nov 01 06:10:26.019922 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:26.019831    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60bdc2d2-b519-4565-8375-61a5254e4f54-workload-socket" pod "60bdc2d2-b519-4565-8375-61a5254e4f54" (UID: "60bdc2d2-b519-4565-8375-61a5254e4f54"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:10:26.019922 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:26.019864    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60bdc2d2-b519-4565-8375-61a5254e4f54-istiod-ca-cert" pod "60bdc2d2-b519-4565-8375-61a5254e4f54" (UID: "60bdc2d2-b519-4565-8375-61a5254e4f54"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:10:26.020142 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:26.019952    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60bdc2d2-b519-4565-8375-61a5254e4f54-workload-certs" pod "60bdc2d2-b519-4565-8375-61a5254e4f54" (UID: "60bdc2d2-b519-4565-8375-61a5254e4f54"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:10:26.020142 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:26.020025    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60bdc2d2-b519-4565-8375-61a5254e4f54-istio-data" pod "60bdc2d2-b519-4565-8375-61a5254e4f54" (UID: "60bdc2d2-b519-4565-8375-61a5254e4f54"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:10:26.020142 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:26.020093    5689 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60bdc2d2-b519-4565-8375-61a5254e4f54-credential-socket\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:10:26.020142 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:26.020108    5689 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60bdc2d2-b519-4565-8375-61a5254e4f54-istiod-ca-cert\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:10:26.020142 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:26.020120    5689 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60bdc2d2-b519-4565-8375-61a5254e4f54-workload-certs\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:10:26.020142 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:26.020130    5689 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60bdc2d2-b519-4565-8375-61a5254e4f54-workload-socket\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:10:26.020142 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:26.020139    5689 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60bdc2d2-b519-4565-8375-61a5254e4f54-istio-data\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:10:26.021874 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:26.021844    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/60bdc2d2-b519-4565-8375-61a5254e4f54-istio-podinfo" pod "60bdc2d2-b519-4565-8375-61a5254e4f54" (UID: "60bdc2d2-b519-4565-8375-61a5254e4f54"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:10:26.021974 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:26.021876    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60bdc2d2-b519-4565-8375-61a5254e4f54-istio-token" pod "60bdc2d2-b519-4565-8375-61a5254e4f54" (UID: "60bdc2d2-b519-4565-8375-61a5254e4f54"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:10:26.021974 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:26.021938    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60bdc2d2-b519-4565-8375-61a5254e4f54-kube-api-access-pj46x" pod "60bdc2d2-b519-4565-8375-61a5254e4f54" (UID: "60bdc2d2-b519-4565-8375-61a5254e4f54"). InnerVolumeSpecName "kube-api-access-pj46x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:10:26.022101 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:26.022081    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60bdc2d2-b519-4565-8375-61a5254e4f54-istio-envoy" pod "60bdc2d2-b519-4565-8375-61a5254e4f54" (UID: "60bdc2d2-b519-4565-8375-61a5254e4f54"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:10:26.121438 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:26.121408    5689 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60bdc2d2-b519-4565-8375-61a5254e4f54-istio-envoy\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:10:26.121438 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:26.121435    5689 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60bdc2d2-b519-4565-8375-61a5254e4f54-istio-podinfo\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:10:26.121638 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:26.121445    5689 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60bdc2d2-b519-4565-8375-61a5254e4f54-istio-token\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:10:26.121638 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:26.121508    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pj46x\" (UniqueName: \"kubernetes.io/projected/60bdc2d2-b519-4565-8375-61a5254e4f54-kube-api-access-pj46x\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:10:26.257702 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:26.257669    5689 generic.go:356] "Generic (PLEG): container finished" podID="60bdc2d2-b519-4565-8375-61a5254e4f54" containerID="f92e65f201136a6cb9dabf193975702518c2b6f9ac3cbb395125e99e329e17e9" exitCode=0
Nov 01 06:10:26.257823 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:26.257729    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3951eb77-openshift-default-85859ff58c-gf998" event={"ID":"60bdc2d2-b519-4565-8375-61a5254e4f54","Type":"ContainerDied","Data":"f92e65f201136a6cb9dabf193975702518c2b6f9ac3cbb395125e99e329e17e9"}
Nov 01 06:10:26.257823 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:26.257797    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3951eb77-openshift-default-85859ff58c-gf998" event={"ID":"60bdc2d2-b519-4565-8375-61a5254e4f54","Type":"ContainerDied","Data":"0fc92f6fea1780244eff84288ab0e672d3d8bac633001027c20ecf4950118a20"}
Nov 01 06:10:26.257823 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:26.257731    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3951eb77-openshift-default-85859ff58c-gf998"
Nov 01 06:10:26.257823 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:26.257820    5689 scope.go:122] "RemoveContainer" containerID="f92e65f201136a6cb9dabf193975702518c2b6f9ac3cbb395125e99e329e17e9"
Nov 01 06:10:26.266241 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:26.266224    5689 scope.go:122] "RemoveContainer" containerID="e3ea866f2c43d4a42da2a6fc43d03f715a9d1109a48a7a2523ea7625b1acd0b7"
Nov 01 06:10:26.273730 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:26.273712    5689 scope.go:122] "RemoveContainer" containerID="f92e65f201136a6cb9dabf193975702518c2b6f9ac3cbb395125e99e329e17e9"
Nov 01 06:10:26.273973 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:10:26.273953    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f92e65f201136a6cb9dabf193975702518c2b6f9ac3cbb395125e99e329e17e9\": container with ID starting with f92e65f201136a6cb9dabf193975702518c2b6f9ac3cbb395125e99e329e17e9 not found: ID does not exist" containerID="f92e65f201136a6cb9dabf193975702518c2b6f9ac3cbb395125e99e329e17e9"
Nov 01 06:10:26.274050 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:26.273985    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f92e65f201136a6cb9dabf193975702518c2b6f9ac3cbb395125e99e329e17e9"} err="failed to get container status \"f92e65f201136a6cb9dabf193975702518c2b6f9ac3cbb395125e99e329e17e9\": rpc error: code = NotFound desc = could not find container \"f92e65f201136a6cb9dabf193975702518c2b6f9ac3cbb395125e99e329e17e9\": container with ID starting with f92e65f201136a6cb9dabf193975702518c2b6f9ac3cbb395125e99e329e17e9 not found: ID does not exist"
Nov 01 06:10:26.274050 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:26.274009    5689 scope.go:122] "RemoveContainer" containerID="e3ea866f2c43d4a42da2a6fc43d03f715a9d1109a48a7a2523ea7625b1acd0b7"
Nov 01 06:10:26.274270 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:10:26.274252    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3ea866f2c43d4a42da2a6fc43d03f715a9d1109a48a7a2523ea7625b1acd0b7\": container with ID starting with e3ea866f2c43d4a42da2a6fc43d03f715a9d1109a48a7a2523ea7625b1acd0b7 not found: ID does not exist" containerID="e3ea866f2c43d4a42da2a6fc43d03f715a9d1109a48a7a2523ea7625b1acd0b7"
Nov 01 06:10:26.274321 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:26.274278    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3ea866f2c43d4a42da2a6fc43d03f715a9d1109a48a7a2523ea7625b1acd0b7"} err="failed to get container status \"e3ea866f2c43d4a42da2a6fc43d03f715a9d1109a48a7a2523ea7625b1acd0b7\": rpc error: code = NotFound desc = could not find container \"e3ea866f2c43d4a42da2a6fc43d03f715a9d1109a48a7a2523ea7625b1acd0b7\": container with ID starting with e3ea866f2c43d4a42da2a6fc43d03f715a9d1109a48a7a2523ea7625b1acd0b7 not found: ID does not exist"
Nov 01 06:10:26.421153 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:26.421123    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3951eb77-openshift-default-85859ff58c-gf998"]
Nov 01 06:10:26.424850 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:26.424826    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-3951eb77-openshift-default-85859ff58c-gf998"]
Nov 01 06:10:27.163768 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:27.163737    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-74b5bc6fff-gmvbm"]
Nov 01 06:10:27.164148 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:27.164073    5689 cpu_manager.go:414] "containerMap: removing container" podUID="eb96ab56-077e-471e-b64e-e336783aaa29" containerName="payload-processing"
Nov 01 06:10:27.164148 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:27.164084    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb96ab56-077e-471e-b64e-e336783aaa29" containerName="payload-processing"
Nov 01 06:10:27.164148 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:27.164092    5689 cpu_manager.go:414] "containerMap: removing container" podUID="c9705308-9add-4b52-86fb-6d5e4b3206c1" containerName="maas-api"
Nov 01 06:10:27.164148 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:27.164097    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9705308-9add-4b52-86fb-6d5e4b3206c1" containerName="maas-api"
Nov 01 06:10:27.164148 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:27.164115    5689 cpu_manager.go:414] "containerMap: removing container" podUID="494650d5-0b43-40d3-b4f0-125df553c510" containerName="payload-pre-processing"
Nov 01 06:10:27.164148 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:27.164121    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="494650d5-0b43-40d3-b4f0-125df553c510" containerName="payload-pre-processing"
Nov 01 06:10:27.164148 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:27.164130    5689 cpu_manager.go:414] "containerMap: removing container" podUID="60bdc2d2-b519-4565-8375-61a5254e4f54" containerName="istio-proxy"
Nov 01 06:10:27.164148 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:27.164134    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60bdc2d2-b519-4565-8375-61a5254e4f54" containerName="istio-proxy"
Nov 01 06:10:27.164148 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:27.164146    5689 cpu_manager.go:414] "containerMap: removing container" podUID="60bdc2d2-b519-4565-8375-61a5254e4f54" containerName="istio-proxy"
Nov 01 06:10:27.164148 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:27.164150    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60bdc2d2-b519-4565-8375-61a5254e4f54" containerName="istio-proxy"
Nov 01 06:10:27.164477 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:27.164204    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="494650d5-0b43-40d3-b4f0-125df553c510" containerName="payload-pre-processing"
Nov 01 06:10:27.164477 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:27.164214    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="60bdc2d2-b519-4565-8375-61a5254e4f54" containerName="istio-proxy"
Nov 01 06:10:27.164477 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:27.164222    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="60bdc2d2-b519-4565-8375-61a5254e4f54" containerName="istio-proxy"
Nov 01 06:10:27.164477 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:27.164230    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb96ab56-077e-471e-b64e-e336783aaa29" containerName="payload-processing"
Nov 01 06:10:27.164477 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:27.164236    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9705308-9add-4b52-86fb-6d5e4b3206c1" containerName="maas-api"
Nov 01 06:10:27.168662 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:27.168642    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-74b5bc6fff-gmvbm"
Nov 01 06:10:27.220763 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:27.220738    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 01 06:10:27.220883 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:27.220861    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 01 06:10:27.220883 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:27.220862    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-v68zh\""
Nov 01 06:10:27.220978 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:27.220912    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 01 06:10:27.271146 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:27.271121    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-74b5bc6fff-gmvbm"]
Nov 01 06:10:27.332558 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:27.332527    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/88c00f37-4dbe-47fb-af14-17cd3e38e151-webhook-cert\") pod \"maas-controller-74b5bc6fff-gmvbm\" (UID: \"88c00f37-4dbe-47fb-af14-17cd3e38e151\") " pod="opendatahub/maas-controller-74b5bc6fff-gmvbm"
Nov 01 06:10:27.332687 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:27.332642    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lx6b\" (UniqueName: \"kubernetes.io/projected/88c00f37-4dbe-47fb-af14-17cd3e38e151-kube-api-access-2lx6b\") pod \"maas-controller-74b5bc6fff-gmvbm\" (UID: \"88c00f37-4dbe-47fb-af14-17cd3e38e151\") " pod="opendatahub/maas-controller-74b5bc6fff-gmvbm"
Nov 01 06:10:27.332734 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:27.332711    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88c00f37-4dbe-47fb-af14-17cd3e38e151-metrics-certs\") pod \"maas-controller-74b5bc6fff-gmvbm\" (UID: \"88c00f37-4dbe-47fb-af14-17cd3e38e151\") " pod="opendatahub/maas-controller-74b5bc6fff-gmvbm"
Nov 01 06:10:27.434059 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:27.433989    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lx6b\" (UniqueName: \"kubernetes.io/projected/88c00f37-4dbe-47fb-af14-17cd3e38e151-kube-api-access-2lx6b\") pod \"maas-controller-74b5bc6fff-gmvbm\" (UID: \"88c00f37-4dbe-47fb-af14-17cd3e38e151\") " pod="opendatahub/maas-controller-74b5bc6fff-gmvbm"
Nov 01 06:10:27.434059 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:27.434033    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88c00f37-4dbe-47fb-af14-17cd3e38e151-metrics-certs\") pod \"maas-controller-74b5bc6fff-gmvbm\" (UID: \"88c00f37-4dbe-47fb-af14-17cd3e38e151\") " pod="opendatahub/maas-controller-74b5bc6fff-gmvbm"
Nov 01 06:10:27.434226 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:27.434065    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/88c00f37-4dbe-47fb-af14-17cd3e38e151-webhook-cert\") pod \"maas-controller-74b5bc6fff-gmvbm\" (UID: \"88c00f37-4dbe-47fb-af14-17cd3e38e151\") " pod="opendatahub/maas-controller-74b5bc6fff-gmvbm"
Nov 01 06:10:27.436628 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:27.436603    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/88c00f37-4dbe-47fb-af14-17cd3e38e151-webhook-cert\") pod \"maas-controller-74b5bc6fff-gmvbm\" (UID: \"88c00f37-4dbe-47fb-af14-17cd3e38e151\") " pod="opendatahub/maas-controller-74b5bc6fff-gmvbm"
Nov 01 06:10:27.436740 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:27.436704    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88c00f37-4dbe-47fb-af14-17cd3e38e151-metrics-certs\") pod \"maas-controller-74b5bc6fff-gmvbm\" (UID: \"88c00f37-4dbe-47fb-af14-17cd3e38e151\") " pod="opendatahub/maas-controller-74b5bc6fff-gmvbm"
Nov 01 06:10:27.508921 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:27.508892    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lx6b\" (UniqueName: \"kubernetes.io/projected/88c00f37-4dbe-47fb-af14-17cd3e38e151-kube-api-access-2lx6b\") pod \"maas-controller-74b5bc6fff-gmvbm\" (UID: \"88c00f37-4dbe-47fb-af14-17cd3e38e151\") " pod="opendatahub/maas-controller-74b5bc6fff-gmvbm"
Nov 01 06:10:27.781198 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:27.781120    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-74b5bc6fff-gmvbm"
Nov 01 06:10:28.066112 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:28.066039    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60bdc2d2-b519-4565-8375-61a5254e4f54" path="/var/lib/kubelet/pods/60bdc2d2-b519-4565-8375-61a5254e4f54/volumes"
Nov 01 06:10:28.157754 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:28.157724    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-74b5bc6fff-gmvbm"]
Nov 01 06:10:28.189220 ip-10-0-129-92 kubenswrapper[5689]: W1101 06:10:28.189192    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88c00f37_4dbe_47fb_af14_17cd3e38e151.slice/crio-d008bc6b6f3a116136f21cc4e375c5c4ab01aeb6a13e1347d1cf889fa7d3919b WatchSource:0}: Error finding container d008bc6b6f3a116136f21cc4e375c5c4ab01aeb6a13e1347d1cf889fa7d3919b: Status 404 returned error can't find the container with id d008bc6b6f3a116136f21cc4e375c5c4ab01aeb6a13e1347d1cf889fa7d3919b
Nov 01 06:10:28.268707 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:28.268681    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-74b5bc6fff-gmvbm" event={"ID":"88c00f37-4dbe-47fb-af14-17cd3e38e151","Type":"ContainerStarted","Data":"d008bc6b6f3a116136f21cc4e375c5c4ab01aeb6a13e1347d1cf889fa7d3919b"}
Nov 01 06:10:30.068242 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:10:30.068210    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 06:10:30.279405 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:30.279369    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-74b5bc6fff-gmvbm" event={"ID":"88c00f37-4dbe-47fb-af14-17cd3e38e151","Type":"ContainerStarted","Data":"042065f4adb574cf75129981b6b7d5e1186ae831825bc4c8b0faa91a18f95f3f"}
Nov 01 06:10:30.279613 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:30.279435    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-74b5bc6fff-gmvbm"
Nov 01 06:10:30.396835 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:30.396765    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-74b5bc6fff-gmvbm" podStartSLOduration=1.479418253 podStartE2EDuration="3.396747176s" totalImagesPullingTime="1.917328923s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:10:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:10:30.396508692 +0000 UTC m=+4008.467305415" watchObservedRunningTime="2026-11-01 06:10:30.396747176 +0000 UTC m=+4008.467543897"
Nov 01 06:10:41.285767 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:41.285740    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-74b5bc6fff-gmvbm"
Nov 01 06:10:49.790430 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:49.790390    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-w2nmc"]
Nov 01 06:10:49.794383 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:49.794362    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-w2nmc"
Nov 01 06:10:49.823241 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:49.823216    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6319ad1-4be9-44bb-898a-139e04529f67-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-w2nmc\" (UID: \"e6319ad1-4be9-44bb-898a-139e04529f67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-w2nmc"
Nov 01 06:10:49.823346 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:49.823258    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6319ad1-4be9-44bb-898a-139e04529f67-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-w2nmc\" (UID: \"e6319ad1-4be9-44bb-898a-139e04529f67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-w2nmc"
Nov 01 06:10:49.823346 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:49.823286    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6319ad1-4be9-44bb-898a-139e04529f67-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-w2nmc\" (UID: \"e6319ad1-4be9-44bb-898a-139e04529f67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-w2nmc"
Nov 01 06:10:49.823435 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:49.823370    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6319ad1-4be9-44bb-898a-139e04529f67-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-w2nmc\" (UID: \"e6319ad1-4be9-44bb-898a-139e04529f67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-w2nmc"
Nov 01 06:10:49.823435 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:49.823401    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnlx5\" (UniqueName: \"kubernetes.io/projected/e6319ad1-4be9-44bb-898a-139e04529f67-kube-api-access-nnlx5\") pod \"maas-default-gateway-openshift-default-58d898bd55-w2nmc\" (UID: \"e6319ad1-4be9-44bb-898a-139e04529f67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-w2nmc"
Nov 01 06:10:49.823551 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:49.823530    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6319ad1-4be9-44bb-898a-139e04529f67-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-w2nmc\" (UID: \"e6319ad1-4be9-44bb-898a-139e04529f67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-w2nmc"
Nov 01 06:10:49.823607 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:49.823578    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6319ad1-4be9-44bb-898a-139e04529f67-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-w2nmc\" (UID: \"e6319ad1-4be9-44bb-898a-139e04529f67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-w2nmc"
Nov 01 06:10:49.823646 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:49.823606    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6319ad1-4be9-44bb-898a-139e04529f67-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-w2nmc\" (UID: \"e6319ad1-4be9-44bb-898a-139e04529f67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-w2nmc"
Nov 01 06:10:49.823701 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:49.823655    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6319ad1-4be9-44bb-898a-139e04529f67-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-w2nmc\" (UID: \"e6319ad1-4be9-44bb-898a-139e04529f67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-w2nmc"
Nov 01 06:10:49.839537 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:49.839511    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c6zlb"]
Nov 01 06:10:49.842970 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:49.842953    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c6zlb"
Nov 01 06:10:49.846575 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:49.846548    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-8hgdn\""
Nov 01 06:10:49.924433 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:49.924400    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnlx5\" (UniqueName: \"kubernetes.io/projected/e6319ad1-4be9-44bb-898a-139e04529f67-kube-api-access-nnlx5\") pod \"maas-default-gateway-openshift-default-58d898bd55-w2nmc\" (UID: \"e6319ad1-4be9-44bb-898a-139e04529f67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-w2nmc"
Nov 01 06:10:49.924612 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:49.924439    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6319ad1-4be9-44bb-898a-139e04529f67-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-w2nmc\" (UID: \"e6319ad1-4be9-44bb-898a-139e04529f67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-w2nmc"
Nov 01 06:10:49.924612 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:49.924491    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frffr\" (UniqueName: \"kubernetes.io/projected/d1d4ddb1-fd17-4ac1-ada4-f785008a0913-kube-api-access-frffr\") pod \"maas-default-gateway-openshift-default-58d898bd55-c6zlb\" (UID: \"d1d4ddb1-fd17-4ac1-ada4-f785008a0913\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c6zlb"
Nov 01 06:10:49.924612 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:49.924516    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1d4ddb1-fd17-4ac1-ada4-f785008a0913-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-c6zlb\" (UID: \"d1d4ddb1-fd17-4ac1-ada4-f785008a0913\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c6zlb"
Nov 01 06:10:49.924612 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:49.924544    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6319ad1-4be9-44bb-898a-139e04529f67-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-w2nmc\" (UID: \"e6319ad1-4be9-44bb-898a-139e04529f67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-w2nmc"
Nov 01 06:10:49.924859 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:49.924690    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6319ad1-4be9-44bb-898a-139e04529f67-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-w2nmc\" (UID: \"e6319ad1-4be9-44bb-898a-139e04529f67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-w2nmc"
Nov 01 06:10:49.924859 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:49.924745    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1d4ddb1-fd17-4ac1-ada4-f785008a0913-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-c6zlb\" (UID: \"d1d4ddb1-fd17-4ac1-ada4-f785008a0913\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c6zlb"
Nov 01 06:10:49.924859 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:49.924778    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6319ad1-4be9-44bb-898a-139e04529f67-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-w2nmc\" (UID: \"e6319ad1-4be9-44bb-898a-139e04529f67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-w2nmc"
Nov 01 06:10:49.924859 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:49.924807    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1d4ddb1-fd17-4ac1-ada4-f785008a0913-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-c6zlb\" (UID: \"d1d4ddb1-fd17-4ac1-ada4-f785008a0913\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c6zlb"
Nov 01 06:10:49.925059 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:49.924914    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6319ad1-4be9-44bb-898a-139e04529f67-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-w2nmc\" (UID: \"e6319ad1-4be9-44bb-898a-139e04529f67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-w2nmc"
Nov 01 06:10:49.925059 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:49.924939    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6319ad1-4be9-44bb-898a-139e04529f67-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-w2nmc\" (UID: \"e6319ad1-4be9-44bb-898a-139e04529f67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-w2nmc"
Nov 01 06:10:49.925059 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:49.924955    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6319ad1-4be9-44bb-898a-139e04529f67-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-w2nmc\" (UID: \"e6319ad1-4be9-44bb-898a-139e04529f67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-w2nmc"
Nov 01 06:10:49.925059 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:49.924991    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1d4ddb1-fd17-4ac1-ada4-f785008a0913-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-c6zlb\" (UID: \"d1d4ddb1-fd17-4ac1-ada4-f785008a0913\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c6zlb"
Nov 01 06:10:49.925059 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:49.925010    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6319ad1-4be9-44bb-898a-139e04529f67-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-w2nmc\" (UID: \"e6319ad1-4be9-44bb-898a-139e04529f67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-w2nmc"
Nov 01 06:10:49.925059 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:49.925023    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1d4ddb1-fd17-4ac1-ada4-f785008a0913-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-c6zlb\" (UID: \"d1d4ddb1-fd17-4ac1-ada4-f785008a0913\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c6zlb"
Nov 01 06:10:49.925305 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:49.925152    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1d4ddb1-fd17-4ac1-ada4-f785008a0913-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-c6zlb\" (UID: \"d1d4ddb1-fd17-4ac1-ada4-f785008a0913\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c6zlb"
Nov 01 06:10:49.925305 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:49.925217    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6319ad1-4be9-44bb-898a-139e04529f67-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-w2nmc\" (UID: \"e6319ad1-4be9-44bb-898a-139e04529f67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-w2nmc"
Nov 01 06:10:49.925305 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:49.925246    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6319ad1-4be9-44bb-898a-139e04529f67-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-w2nmc\" (UID: \"e6319ad1-4be9-44bb-898a-139e04529f67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-w2nmc"
Nov 01 06:10:49.925305 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:49.925272    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1d4ddb1-fd17-4ac1-ada4-f785008a0913-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-c6zlb\" (UID: \"d1d4ddb1-fd17-4ac1-ada4-f785008a0913\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c6zlb"
Nov 01 06:10:49.925305 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:49.925294    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1d4ddb1-fd17-4ac1-ada4-f785008a0913-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-c6zlb\" (UID: \"d1d4ddb1-fd17-4ac1-ada4-f785008a0913\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c6zlb"
Nov 01 06:10:49.925756 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:49.925316    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6319ad1-4be9-44bb-898a-139e04529f67-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-w2nmc\" (UID: \"e6319ad1-4be9-44bb-898a-139e04529f67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-w2nmc"
Nov 01 06:10:49.925756 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:49.925616    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6319ad1-4be9-44bb-898a-139e04529f67-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-w2nmc\" (UID: \"e6319ad1-4be9-44bb-898a-139e04529f67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-w2nmc"
Nov 01 06:10:49.925876 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:49.925862    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6319ad1-4be9-44bb-898a-139e04529f67-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-w2nmc\" (UID: \"e6319ad1-4be9-44bb-898a-139e04529f67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-w2nmc"
Nov 01 06:10:49.927165 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:49.927146    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6319ad1-4be9-44bb-898a-139e04529f67-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-w2nmc\" (UID: \"e6319ad1-4be9-44bb-898a-139e04529f67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-w2nmc"
Nov 01 06:10:49.927428 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:49.927408    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6319ad1-4be9-44bb-898a-139e04529f67-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-w2nmc\" (UID: \"e6319ad1-4be9-44bb-898a-139e04529f67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-w2nmc"
Nov 01 06:10:49.937390 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:49.937364    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c6zlb"]
Nov 01 06:10:49.985438 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:49.985403    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6319ad1-4be9-44bb-898a-139e04529f67-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-w2nmc\" (UID: \"e6319ad1-4be9-44bb-898a-139e04529f67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-w2nmc"
Nov 01 06:10:49.985619 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:49.985562    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnlx5\" (UniqueName: \"kubernetes.io/projected/e6319ad1-4be9-44bb-898a-139e04529f67-kube-api-access-nnlx5\") pod \"maas-default-gateway-openshift-default-58d898bd55-w2nmc\" (UID: \"e6319ad1-4be9-44bb-898a-139e04529f67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-w2nmc"
Nov 01 06:10:50.002752 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:50.002725    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-w2nmc"]
Nov 01 06:10:50.026693 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:50.026656    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1d4ddb1-fd17-4ac1-ada4-f785008a0913-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-c6zlb\" (UID: \"d1d4ddb1-fd17-4ac1-ada4-f785008a0913\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c6zlb"
Nov 01 06:10:50.026826 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:50.026723    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1d4ddb1-fd17-4ac1-ada4-f785008a0913-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-c6zlb\" (UID: \"d1d4ddb1-fd17-4ac1-ada4-f785008a0913\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c6zlb"
Nov 01 06:10:50.026826 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:50.026754    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1d4ddb1-fd17-4ac1-ada4-f785008a0913-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-c6zlb\" (UID: \"d1d4ddb1-fd17-4ac1-ada4-f785008a0913\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c6zlb"
Nov 01 06:10:50.026826 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:50.026781    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1d4ddb1-fd17-4ac1-ada4-f785008a0913-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-c6zlb\" (UID: \"d1d4ddb1-fd17-4ac1-ada4-f785008a0913\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c6zlb"
Nov 01 06:10:50.026826 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:50.026821    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1d4ddb1-fd17-4ac1-ada4-f785008a0913-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-c6zlb\" (UID: \"d1d4ddb1-fd17-4ac1-ada4-f785008a0913\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c6zlb"
Nov 01 06:10:50.027017 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:50.026849    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1d4ddb1-fd17-4ac1-ada4-f785008a0913-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-c6zlb\" (UID: \"d1d4ddb1-fd17-4ac1-ada4-f785008a0913\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c6zlb"
Nov 01 06:10:50.027017 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:50.026924    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frffr\" (UniqueName: \"kubernetes.io/projected/d1d4ddb1-fd17-4ac1-ada4-f785008a0913-kube-api-access-frffr\") pod \"maas-default-gateway-openshift-default-58d898bd55-c6zlb\" (UID: \"d1d4ddb1-fd17-4ac1-ada4-f785008a0913\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c6zlb"
Nov 01 06:10:50.027017 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:50.026958    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1d4ddb1-fd17-4ac1-ada4-f785008a0913-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-c6zlb\" (UID: \"d1d4ddb1-fd17-4ac1-ada4-f785008a0913\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c6zlb"
Nov 01 06:10:50.027207 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:50.027049    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1d4ddb1-fd17-4ac1-ada4-f785008a0913-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-c6zlb\" (UID: \"d1d4ddb1-fd17-4ac1-ada4-f785008a0913\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c6zlb"
Nov 01 06:10:50.027207 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:50.027153    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1d4ddb1-fd17-4ac1-ada4-f785008a0913-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-c6zlb\" (UID: \"d1d4ddb1-fd17-4ac1-ada4-f785008a0913\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c6zlb"
Nov 01 06:10:50.027322 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:50.027206    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1d4ddb1-fd17-4ac1-ada4-f785008a0913-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-c6zlb\" (UID: \"d1d4ddb1-fd17-4ac1-ada4-f785008a0913\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c6zlb"
Nov 01 06:10:50.027322 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:50.027245    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1d4ddb1-fd17-4ac1-ada4-f785008a0913-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-c6zlb\" (UID: \"d1d4ddb1-fd17-4ac1-ada4-f785008a0913\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c6zlb"
Nov 01 06:10:50.027616 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:50.027588    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1d4ddb1-fd17-4ac1-ada4-f785008a0913-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-c6zlb\" (UID: \"d1d4ddb1-fd17-4ac1-ada4-f785008a0913\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c6zlb"
Nov 01 06:10:50.032483 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:50.028213    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1d4ddb1-fd17-4ac1-ada4-f785008a0913-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-c6zlb\" (UID: \"d1d4ddb1-fd17-4ac1-ada4-f785008a0913\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c6zlb"
Nov 01 06:10:50.032483 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:50.029678    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1d4ddb1-fd17-4ac1-ada4-f785008a0913-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-c6zlb\" (UID: \"d1d4ddb1-fd17-4ac1-ada4-f785008a0913\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c6zlb"
Nov 01 06:10:50.032483 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:50.030091    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1d4ddb1-fd17-4ac1-ada4-f785008a0913-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-c6zlb\" (UID: \"d1d4ddb1-fd17-4ac1-ada4-f785008a0913\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c6zlb"
Nov 01 06:10:50.089400 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:50.089330    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1d4ddb1-fd17-4ac1-ada4-f785008a0913-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-c6zlb\" (UID: \"d1d4ddb1-fd17-4ac1-ada4-f785008a0913\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c6zlb"
Nov 01 06:10:50.089580 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:50.089527    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frffr\" (UniqueName: \"kubernetes.io/projected/d1d4ddb1-fd17-4ac1-ada4-f785008a0913-kube-api-access-frffr\") pod \"maas-default-gateway-openshift-default-58d898bd55-c6zlb\" (UID: \"d1d4ddb1-fd17-4ac1-ada4-f785008a0913\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c6zlb"
Nov 01 06:10:50.107408 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:50.107383    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-w2nmc"
Nov 01 06:10:50.155728 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:50.155506    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c6zlb"
Nov 01 06:10:50.534920 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:50.534889    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-w2nmc"]
Nov 01 06:10:50.538373 ip-10-0-129-92 kubenswrapper[5689]: W1101 06:10:50.538344    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6319ad1_4be9_44bb_898a_139e04529f67.slice/crio-a0e52769a23df4b9c292da5f741174d6655be4e11c92d8c705218289b710326e WatchSource:0}: Error finding container a0e52769a23df4b9c292da5f741174d6655be4e11c92d8c705218289b710326e: Status 404 returned error can't find the container with id a0e52769a23df4b9c292da5f741174d6655be4e11c92d8c705218289b710326e
Nov 01 06:10:50.540509 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:50.540480    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 06:10:50.540592 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:50.540539    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 06:10:50.547318 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:50.541002    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 06:10:50.592785 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:50.592758    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c6zlb"]
Nov 01 06:10:50.595004 ip-10-0-129-92 kubenswrapper[5689]: W1101 06:10:50.594982    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1d4ddb1_fd17_4ac1_ada4_f785008a0913.slice/crio-35153fc9ff729fd3a5208d48a4d2c0cee670aed8ccf9d1956dfb7c675d670a3f WatchSource:0}: Error finding container 35153fc9ff729fd3a5208d48a4d2c0cee670aed8ccf9d1956dfb7c675d670a3f: Status 404 returned error can't find the container with id 35153fc9ff729fd3a5208d48a4d2c0cee670aed8ccf9d1956dfb7c675d670a3f
Nov 01 06:10:50.597404 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:50.597368    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 06:10:50.597542 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:50.597519    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 06:10:50.597592 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:50.597576    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 06:10:51.370218 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:51.370188    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c6zlb" event={"ID":"d1d4ddb1-fd17-4ac1-ada4-f785008a0913","Type":"ContainerStarted","Data":"d61fe4ea4a527bb5d0f41fb71f2b0a013c0f5bb00270dae7f50ff87c27fe4fc5"}
Nov 01 06:10:51.370600 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:51.370237    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c6zlb" event={"ID":"d1d4ddb1-fd17-4ac1-ada4-f785008a0913","Type":"ContainerStarted","Data":"35153fc9ff729fd3a5208d48a4d2c0cee670aed8ccf9d1956dfb7c675d670a3f"}
Nov 01 06:10:51.372027 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:51.372005    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-w2nmc" event={"ID":"e6319ad1-4be9-44bb-898a-139e04529f67","Type":"ContainerStarted","Data":"5de4b3ca170a0b3ccc933b30df4417530bf7ff7009731c1a8e9ac4640a5fba74"}
Nov 01 06:10:51.372077 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:51.372044    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-w2nmc" event={"ID":"e6319ad1-4be9-44bb-898a-139e04529f67","Type":"ContainerStarted","Data":"a0e52769a23df4b9c292da5f741174d6655be4e11c92d8c705218289b710326e"}
Nov 01 06:10:51.498479 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:51.498108    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c6zlb" podStartSLOduration=2.498089438 podStartE2EDuration="2.498089438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:10:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:10:51.496243012 +0000 UTC m=+4029.567039735" watchObservedRunningTime="2026-11-01 06:10:51.498089438 +0000 UTC m=+4029.568886160"
Nov 01 06:10:51.624641 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:51.619832    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-w2nmc" podStartSLOduration=2.619809655 podStartE2EDuration="2.619809655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:10:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:10:51.617941061 +0000 UTC m=+4029.688737793" watchObservedRunningTime="2026-11-01 06:10:51.619809655 +0000 UTC m=+4029.690606376"
Nov 01 06:10:52.108477 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:52.107788    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-w2nmc"
Nov 01 06:10:52.124477 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:52.114570    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-w2nmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Nov 01 06:10:52.124477 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:52.114629    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-w2nmc" podUID="e6319ad1-4be9-44bb-898a-139e04529f67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Nov 01 06:10:52.155933 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:52.155903    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c6zlb"
Nov 01 06:10:52.157212 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:52.157189    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-c6zlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body=
Nov 01 06:10:52.157308 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:52.157246    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c6zlb" podUID="d1d4ddb1-fd17-4ac1-ada4-f785008a0913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused"
Nov 01 06:10:53.107922 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:53.107882    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-w2nmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Nov 01 06:10:53.108571 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:53.108539    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-w2nmc" podUID="e6319ad1-4be9-44bb-898a-139e04529f67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Nov 01 06:10:53.155850 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:53.155820    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-c6zlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body=
Nov 01 06:10:53.156002 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:53.155868    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c6zlb" podUID="d1d4ddb1-fd17-4ac1-ada4-f785008a0913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused"
Nov 01 06:10:54.108480 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:54.107829    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-w2nmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Nov 01 06:10:54.108480 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:54.107903    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-w2nmc" podUID="e6319ad1-4be9-44bb-898a-139e04529f67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Nov 01 06:10:54.156839 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:54.156807    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-c6zlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body=
Nov 01 06:10:54.157011 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:54.156854    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c6zlb" podUID="d1d4ddb1-fd17-4ac1-ada4-f785008a0913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused"
Nov 01 06:10:55.107887 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:55.107849    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-w2nmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Nov 01 06:10:55.108219 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:55.108188    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-w2nmc" podUID="e6319ad1-4be9-44bb-898a-139e04529f67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Nov 01 06:10:55.156383 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:55.156355    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-c6zlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body=
Nov 01 06:10:55.156811 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:55.156529    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c6zlb" podUID="d1d4ddb1-fd17-4ac1-ada4-f785008a0913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused"
Nov 01 06:10:56.108484 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:56.107886    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-w2nmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Nov 01 06:10:56.108484 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:56.107952    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-w2nmc" podUID="e6319ad1-4be9-44bb-898a-139e04529f67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Nov 01 06:10:56.156648 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:56.156613    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-c6zlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body=
Nov 01 06:10:56.157036 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:56.156668    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c6zlb" podUID="d1d4ddb1-fd17-4ac1-ada4-f785008a0913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused"
Nov 01 06:10:57.107968 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:57.107929    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-w2nmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Nov 01 06:10:57.108137 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:57.108007    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-w2nmc" podUID="e6319ad1-4be9-44bb-898a-139e04529f67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Nov 01 06:10:57.156538 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:57.156504    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-c6zlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body=
Nov 01 06:10:57.156668 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:57.156563    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c6zlb" podUID="d1d4ddb1-fd17-4ac1-ada4-f785008a0913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused"
Nov 01 06:10:59.108332 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:59.108291    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-w2nmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 06:10:59.108759 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:59.108374    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-w2nmc" podUID="e6319ad1-4be9-44bb-898a-139e04529f67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 06:10:59.157257 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:59.157220    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-c6zlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 06:10:59.157437 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:10:59.157273    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c6zlb" podUID="d1d4ddb1-fd17-4ac1-ada4-f785008a0913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 06:11:00.069029 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:11:00.068993    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 06:11:00.150601 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:00.150568    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-w2nmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 06:11:00.150984 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:00.150613    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-w2nmc" podUID="e6319ad1-4be9-44bb-898a-139e04529f67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 06:11:00.199601 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:00.199552    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-c6zlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 06:11:00.199762 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:00.199618    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c6zlb" podUID="d1d4ddb1-fd17-4ac1-ada4-f785008a0913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 06:11:01.149094 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:01.149050    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-w2nmc"
Nov 01 06:11:01.149424 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:01.149396    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-w2nmc"
Nov 01 06:11:01.150411 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:01.150390    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-w2nmc"
Nov 01 06:11:01.214186 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:01.214155    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-c6zlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 06:11:01.214627 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:01.214202    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c6zlb" podUID="d1d4ddb1-fd17-4ac1-ada4-f785008a0913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 06:11:01.217303 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:01.217281    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c6zlb"
Nov 01 06:11:01.217595 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:01.217577    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c6zlb"
Nov 01 06:11:01.218226 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:01.218205    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c6zlb"
Nov 01 06:11:02.545291 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:02.545260    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:11:02.653518 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:02.653490    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-74b5bc6fff-gmvbm"]
Nov 01 06:11:02.653701 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:02.653668    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-74b5bc6fff-gmvbm" podUID="88c00f37-4dbe-47fb-af14-17cd3e38e151" containerName="manager" containerID="cri-o://042065f4adb574cf75129981b6b7d5e1186ae831825bc4c8b0faa91a18f95f3f" gracePeriod=30
Nov 01 06:11:02.895534 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:02.895511    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-74b5bc6fff-gmvbm"
Nov 01 06:11:02.983388 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:02.983365    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88c00f37-4dbe-47fb-af14-17cd3e38e151-kube-api-access-2lx6b\" (UniqueName: \"kubernetes.io/projected/88c00f37-4dbe-47fb-af14-17cd3e38e151-kube-api-access-2lx6b\") pod \"88c00f37-4dbe-47fb-af14-17cd3e38e151\" (UID: \"88c00f37-4dbe-47fb-af14-17cd3e38e151\") "
Nov 01 06:11:02.983561 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:02.983499    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88c00f37-4dbe-47fb-af14-17cd3e38e151-webhook-cert\" (UniqueName: \"kubernetes.io/secret/88c00f37-4dbe-47fb-af14-17cd3e38e151-webhook-cert\") pod \"88c00f37-4dbe-47fb-af14-17cd3e38e151\" (UID: \"88c00f37-4dbe-47fb-af14-17cd3e38e151\") "
Nov 01 06:11:02.983561 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:02.983535    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88c00f37-4dbe-47fb-af14-17cd3e38e151-metrics-certs\" (UniqueName: \"kubernetes.io/secret/88c00f37-4dbe-47fb-af14-17cd3e38e151-metrics-certs\") pod \"88c00f37-4dbe-47fb-af14-17cd3e38e151\" (UID: \"88c00f37-4dbe-47fb-af14-17cd3e38e151\") "
Nov 01 06:11:02.985764 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:02.985732    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88c00f37-4dbe-47fb-af14-17cd3e38e151-kube-api-access-2lx6b" pod "88c00f37-4dbe-47fb-af14-17cd3e38e151" (UID: "88c00f37-4dbe-47fb-af14-17cd3e38e151"). InnerVolumeSpecName "kube-api-access-2lx6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:11:02.985863 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:02.985757    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88c00f37-4dbe-47fb-af14-17cd3e38e151-metrics-certs" pod "88c00f37-4dbe-47fb-af14-17cd3e38e151" (UID: "88c00f37-4dbe-47fb-af14-17cd3e38e151"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:11:02.985863 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:02.985732    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88c00f37-4dbe-47fb-af14-17cd3e38e151-webhook-cert" pod "88c00f37-4dbe-47fb-af14-17cd3e38e151" (UID: "88c00f37-4dbe-47fb-af14-17cd3e38e151"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:11:03.084440 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:03.084363    5689 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/88c00f37-4dbe-47fb-af14-17cd3e38e151-webhook-cert\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:11:03.084440 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:03.084389    5689 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88c00f37-4dbe-47fb-af14-17cd3e38e151-metrics-certs\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:11:03.084440 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:03.084399    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lx6b\" (UniqueName: \"kubernetes.io/projected/88c00f37-4dbe-47fb-af14-17cd3e38e151-kube-api-access-2lx6b\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:11:03.489193 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:03.489164    5689 generic.go:356] "Generic (PLEG): container finished" podID="88c00f37-4dbe-47fb-af14-17cd3e38e151" containerID="042065f4adb574cf75129981b6b7d5e1186ae831825bc4c8b0faa91a18f95f3f" exitCode=0
Nov 01 06:11:03.489375 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:03.489208    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-74b5bc6fff-gmvbm"
Nov 01 06:11:03.489375 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:03.489250    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-74b5bc6fff-gmvbm" event={"ID":"88c00f37-4dbe-47fb-af14-17cd3e38e151","Type":"ContainerDied","Data":"042065f4adb574cf75129981b6b7d5e1186ae831825bc4c8b0faa91a18f95f3f"}
Nov 01 06:11:03.489375 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:03.489298    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-74b5bc6fff-gmvbm" event={"ID":"88c00f37-4dbe-47fb-af14-17cd3e38e151","Type":"ContainerDied","Data":"d008bc6b6f3a116136f21cc4e375c5c4ab01aeb6a13e1347d1cf889fa7d3919b"}
Nov 01 06:11:03.489375 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:03.489319    5689 scope.go:122] "RemoveContainer" containerID="042065f4adb574cf75129981b6b7d5e1186ae831825bc4c8b0faa91a18f95f3f"
Nov 01 06:11:03.498815 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:03.498797    5689 scope.go:122] "RemoveContainer" containerID="042065f4adb574cf75129981b6b7d5e1186ae831825bc4c8b0faa91a18f95f3f"
Nov 01 06:11:03.499082 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:11:03.499060    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"042065f4adb574cf75129981b6b7d5e1186ae831825bc4c8b0faa91a18f95f3f\": container with ID starting with 042065f4adb574cf75129981b6b7d5e1186ae831825bc4c8b0faa91a18f95f3f not found: ID does not exist" containerID="042065f4adb574cf75129981b6b7d5e1186ae831825bc4c8b0faa91a18f95f3f"
Nov 01 06:11:03.499177 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:03.499087    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"042065f4adb574cf75129981b6b7d5e1186ae831825bc4c8b0faa91a18f95f3f"} err="failed to get container status \"042065f4adb574cf75129981b6b7d5e1186ae831825bc4c8b0faa91a18f95f3f\": rpc error: code = NotFound desc = could not find container \"042065f4adb574cf75129981b6b7d5e1186ae831825bc4c8b0faa91a18f95f3f\": container with ID starting with 042065f4adb574cf75129981b6b7d5e1186ae831825bc4c8b0faa91a18f95f3f not found: ID does not exist"
Nov 01 06:11:03.652643 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:03.652612    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-74b5bc6fff-gmvbm"]
Nov 01 06:11:03.702243 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:03.702213    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-74b5bc6fff-gmvbm"]
Nov 01 06:11:04.066226 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:04.066197    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88c00f37-4dbe-47fb-af14-17cd3e38e151" path="/var/lib/kubelet/pods/88c00f37-4dbe-47fb-af14-17cd3e38e151/volumes"
Nov 01 06:11:04.932612 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:04.932582    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:11:05.055548 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:05.055516    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-74b5bc6fff-6xcrb"]
Nov 01 06:11:05.055945 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:05.055930    5689 cpu_manager.go:414] "containerMap: removing container" podUID="88c00f37-4dbe-47fb-af14-17cd3e38e151" containerName="manager"
Nov 01 06:11:05.055992 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:05.055947    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88c00f37-4dbe-47fb-af14-17cd3e38e151" containerName="manager"
Nov 01 06:11:05.056045 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:05.056025    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="88c00f37-4dbe-47fb-af14-17cd3e38e151" containerName="manager"
Nov 01 06:11:05.061162 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:05.061145    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-74b5bc6fff-6xcrb"
Nov 01 06:11:05.162785 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:05.162755    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 01 06:11:05.162992 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:05.162933    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 01 06:11:05.162992 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:05.162947    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-v68zh\""
Nov 01 06:11:05.163124 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:05.163068    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 01 06:11:05.165167 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:05.165141    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-74b5bc6fff-6xcrb"]
Nov 01 06:11:05.202248 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:05.202179    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55552040-c621-4372-89bd-7e41eb45f41e-metrics-certs\") pod \"maas-controller-74b5bc6fff-6xcrb\" (UID: \"55552040-c621-4372-89bd-7e41eb45f41e\") " pod="opendatahub/maas-controller-74b5bc6fff-6xcrb"
Nov 01 06:11:05.202378 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:05.202346    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gsbs\" (UniqueName: \"kubernetes.io/projected/55552040-c621-4372-89bd-7e41eb45f41e-kube-api-access-7gsbs\") pod \"maas-controller-74b5bc6fff-6xcrb\" (UID: \"55552040-c621-4372-89bd-7e41eb45f41e\") " pod="opendatahub/maas-controller-74b5bc6fff-6xcrb"
Nov 01 06:11:05.202426 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:05.202385    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/55552040-c621-4372-89bd-7e41eb45f41e-webhook-cert\") pod \"maas-controller-74b5bc6fff-6xcrb\" (UID: \"55552040-c621-4372-89bd-7e41eb45f41e\") " pod="opendatahub/maas-controller-74b5bc6fff-6xcrb"
Nov 01 06:11:05.303595 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:05.303568    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gsbs\" (UniqueName: \"kubernetes.io/projected/55552040-c621-4372-89bd-7e41eb45f41e-kube-api-access-7gsbs\") pod \"maas-controller-74b5bc6fff-6xcrb\" (UID: \"55552040-c621-4372-89bd-7e41eb45f41e\") " pod="opendatahub/maas-controller-74b5bc6fff-6xcrb"
Nov 01 06:11:05.303747 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:05.303605    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/55552040-c621-4372-89bd-7e41eb45f41e-webhook-cert\") pod \"maas-controller-74b5bc6fff-6xcrb\" (UID: \"55552040-c621-4372-89bd-7e41eb45f41e\") " pod="opendatahub/maas-controller-74b5bc6fff-6xcrb"
Nov 01 06:11:05.303747 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:05.303663    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55552040-c621-4372-89bd-7e41eb45f41e-metrics-certs\") pod \"maas-controller-74b5bc6fff-6xcrb\" (UID: \"55552040-c621-4372-89bd-7e41eb45f41e\") " pod="opendatahub/maas-controller-74b5bc6fff-6xcrb"
Nov 01 06:11:05.306020 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:05.305991    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/55552040-c621-4372-89bd-7e41eb45f41e-webhook-cert\") pod \"maas-controller-74b5bc6fff-6xcrb\" (UID: \"55552040-c621-4372-89bd-7e41eb45f41e\") " pod="opendatahub/maas-controller-74b5bc6fff-6xcrb"
Nov 01 06:11:05.306121 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:05.306099    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55552040-c621-4372-89bd-7e41eb45f41e-metrics-certs\") pod \"maas-controller-74b5bc6fff-6xcrb\" (UID: \"55552040-c621-4372-89bd-7e41eb45f41e\") " pod="opendatahub/maas-controller-74b5bc6fff-6xcrb"
Nov 01 06:11:05.364302 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:05.364278    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gsbs\" (UniqueName: \"kubernetes.io/projected/55552040-c621-4372-89bd-7e41eb45f41e-kube-api-access-7gsbs\") pod \"maas-controller-74b5bc6fff-6xcrb\" (UID: \"55552040-c621-4372-89bd-7e41eb45f41e\") " pod="opendatahub/maas-controller-74b5bc6fff-6xcrb"
Nov 01 06:11:05.375131 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:05.375110    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-74b5bc6fff-6xcrb"
Nov 01 06:11:05.749219 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:05.749190    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-74b5bc6fff-6xcrb"]
Nov 01 06:11:05.752866 ip-10-0-129-92 kubenswrapper[5689]: W1101 06:11:05.752838    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55552040_c621_4372_89bd_7e41eb45f41e.slice/crio-57560fe91f5eb0824ff26618dbd2fd76ae59391230c4306dc674442f85bb706a WatchSource:0}: Error finding container 57560fe91f5eb0824ff26618dbd2fd76ae59391230c4306dc674442f85bb706a: Status 404 returned error can't find the container with id 57560fe91f5eb0824ff26618dbd2fd76ae59391230c4306dc674442f85bb706a
Nov 01 06:11:06.505146 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:06.505110    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-74b5bc6fff-6xcrb" event={"ID":"55552040-c621-4372-89bd-7e41eb45f41e","Type":"ContainerStarted","Data":"57560fe91f5eb0824ff26618dbd2fd76ae59391230c4306dc674442f85bb706a"}
Nov 01 06:11:07.511016 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:07.510984    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-74b5bc6fff-6xcrb" event={"ID":"55552040-c621-4372-89bd-7e41eb45f41e","Type":"ContainerStarted","Data":"1c89c7b438b747ea567a989266a57cc9fcbfd0d3dc14b20859b7490092d4d1e2"}
Nov 01 06:11:07.511371 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:07.511094    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-74b5bc6fff-6xcrb"
Nov 01 06:11:07.629431 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:07.629382    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-74b5bc6fff-6xcrb" podStartSLOduration=1.8342946329999998 podStartE2EDuration="2.629369024s" totalImagesPullingTime="795.074391ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:11:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:11:07.628285803 +0000 UTC m=+4045.699082525" watchObservedRunningTime="2026-11-01 06:11:07.629369024 +0000 UTC m=+4045.700165743"
Nov 01 06:11:10.603809 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:10.603776    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:11:10.769921 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:10.769888    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:11:16.796653 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:16.796617    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:11:18.517442 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:18.517415    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-74b5bc6fff-6xcrb"
Nov 01 06:11:30.067881 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:11:30.067807    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 06:11:33.570044 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:33.570011    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:11:50.265685 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:50.265652    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cd9pn"]
Nov 01 06:11:50.268904 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:50.268885    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cd9pn"
Nov 01 06:11:50.374019 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:50.373987    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cd9pn"]
Nov 01 06:11:50.403069 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:50.403040    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce6f3d72-c7f9-43a9-83ce-40ab705d092a-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-cd9pn\" (UID: \"ce6f3d72-c7f9-43a9-83ce-40ab705d092a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cd9pn"
Nov 01 06:11:50.403268 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:50.403094    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce6f3d72-c7f9-43a9-83ce-40ab705d092a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-cd9pn\" (UID: \"ce6f3d72-c7f9-43a9-83ce-40ab705d092a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cd9pn"
Nov 01 06:11:50.403268 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:50.403190    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce6f3d72-c7f9-43a9-83ce-40ab705d092a-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-cd9pn\" (UID: \"ce6f3d72-c7f9-43a9-83ce-40ab705d092a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cd9pn"
Nov 01 06:11:50.403353 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:50.403316    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce6f3d72-c7f9-43a9-83ce-40ab705d092a-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-cd9pn\" (UID: \"ce6f3d72-c7f9-43a9-83ce-40ab705d092a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cd9pn"
Nov 01 06:11:50.403353 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:50.403343    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce6f3d72-c7f9-43a9-83ce-40ab705d092a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-cd9pn\" (UID: \"ce6f3d72-c7f9-43a9-83ce-40ab705d092a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cd9pn"
Nov 01 06:11:50.403432 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:50.403417    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce6f3d72-c7f9-43a9-83ce-40ab705d092a-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-cd9pn\" (UID: \"ce6f3d72-c7f9-43a9-83ce-40ab705d092a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cd9pn"
Nov 01 06:11:50.403524 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:50.403510    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce6f3d72-c7f9-43a9-83ce-40ab705d092a-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-cd9pn\" (UID: \"ce6f3d72-c7f9-43a9-83ce-40ab705d092a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cd9pn"
Nov 01 06:11:50.403561 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:50.403549    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jd9h\" (UniqueName: \"kubernetes.io/projected/ce6f3d72-c7f9-43a9-83ce-40ab705d092a-kube-api-access-2jd9h\") pod \"maas-default-gateway-openshift-default-58d898bd55-cd9pn\" (UID: \"ce6f3d72-c7f9-43a9-83ce-40ab705d092a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cd9pn"
Nov 01 06:11:50.403598 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:50.403583    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce6f3d72-c7f9-43a9-83ce-40ab705d092a-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-cd9pn\" (UID: \"ce6f3d72-c7f9-43a9-83ce-40ab705d092a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cd9pn"
Nov 01 06:11:50.504871 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:50.504837    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jd9h\" (UniqueName: \"kubernetes.io/projected/ce6f3d72-c7f9-43a9-83ce-40ab705d092a-kube-api-access-2jd9h\") pod \"maas-default-gateway-openshift-default-58d898bd55-cd9pn\" (UID: \"ce6f3d72-c7f9-43a9-83ce-40ab705d092a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cd9pn"
Nov 01 06:11:50.505045 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:50.504884    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce6f3d72-c7f9-43a9-83ce-40ab705d092a-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-cd9pn\" (UID: \"ce6f3d72-c7f9-43a9-83ce-40ab705d092a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cd9pn"
Nov 01 06:11:50.505045 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:50.504912    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce6f3d72-c7f9-43a9-83ce-40ab705d092a-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-cd9pn\" (UID: \"ce6f3d72-c7f9-43a9-83ce-40ab705d092a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cd9pn"
Nov 01 06:11:50.505045 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:50.504974    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce6f3d72-c7f9-43a9-83ce-40ab705d092a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-cd9pn\" (UID: \"ce6f3d72-c7f9-43a9-83ce-40ab705d092a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cd9pn"
Nov 01 06:11:50.505045 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:50.504997    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce6f3d72-c7f9-43a9-83ce-40ab705d092a-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-cd9pn\" (UID: \"ce6f3d72-c7f9-43a9-83ce-40ab705d092a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cd9pn"
Nov 01 06:11:50.505045 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:50.505032    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce6f3d72-c7f9-43a9-83ce-40ab705d092a-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-cd9pn\" (UID: \"ce6f3d72-c7f9-43a9-83ce-40ab705d092a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cd9pn"
Nov 01 06:11:50.505370 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:50.505055    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce6f3d72-c7f9-43a9-83ce-40ab705d092a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-cd9pn\" (UID: \"ce6f3d72-c7f9-43a9-83ce-40ab705d092a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cd9pn"
Nov 01 06:11:50.505370 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:50.505082    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce6f3d72-c7f9-43a9-83ce-40ab705d092a-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-cd9pn\" (UID: \"ce6f3d72-c7f9-43a9-83ce-40ab705d092a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cd9pn"
Nov 01 06:11:50.505370 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:50.505167    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce6f3d72-c7f9-43a9-83ce-40ab705d092a-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-cd9pn\" (UID: \"ce6f3d72-c7f9-43a9-83ce-40ab705d092a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cd9pn"
Nov 01 06:11:50.505590 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:50.505430    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce6f3d72-c7f9-43a9-83ce-40ab705d092a-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-cd9pn\" (UID: \"ce6f3d72-c7f9-43a9-83ce-40ab705d092a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cd9pn"
Nov 01 06:11:50.505590 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:50.505509    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce6f3d72-c7f9-43a9-83ce-40ab705d092a-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-cd9pn\" (UID: \"ce6f3d72-c7f9-43a9-83ce-40ab705d092a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cd9pn"
Nov 01 06:11:50.505696 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:50.505608    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce6f3d72-c7f9-43a9-83ce-40ab705d092a-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-cd9pn\" (UID: \"ce6f3d72-c7f9-43a9-83ce-40ab705d092a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cd9pn"
Nov 01 06:11:50.505696 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:50.505657    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce6f3d72-c7f9-43a9-83ce-40ab705d092a-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-cd9pn\" (UID: \"ce6f3d72-c7f9-43a9-83ce-40ab705d092a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cd9pn"
Nov 01 06:11:50.506100 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:50.506076    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce6f3d72-c7f9-43a9-83ce-40ab705d092a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-cd9pn\" (UID: \"ce6f3d72-c7f9-43a9-83ce-40ab705d092a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cd9pn"
Nov 01 06:11:50.507391 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:50.507369    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce6f3d72-c7f9-43a9-83ce-40ab705d092a-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-cd9pn\" (UID: \"ce6f3d72-c7f9-43a9-83ce-40ab705d092a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cd9pn"
Nov 01 06:11:50.507662 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:50.507644    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce6f3d72-c7f9-43a9-83ce-40ab705d092a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-cd9pn\" (UID: \"ce6f3d72-c7f9-43a9-83ce-40ab705d092a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cd9pn"
Nov 01 06:11:50.565402 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:50.565356    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce6f3d72-c7f9-43a9-83ce-40ab705d092a-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-cd9pn\" (UID: \"ce6f3d72-c7f9-43a9-83ce-40ab705d092a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cd9pn"
Nov 01 06:11:50.565707 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:50.565686    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jd9h\" (UniqueName: \"kubernetes.io/projected/ce6f3d72-c7f9-43a9-83ce-40ab705d092a-kube-api-access-2jd9h\") pod \"maas-default-gateway-openshift-default-58d898bd55-cd9pn\" (UID: \"ce6f3d72-c7f9-43a9-83ce-40ab705d092a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cd9pn"
Nov 01 06:11:50.582072 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:50.582049    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cd9pn"
Nov 01 06:11:51.008923 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:51.008893    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cd9pn"]
Nov 01 06:11:51.012165 ip-10-0-129-92 kubenswrapper[5689]: W1101 06:11:51.012136    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce6f3d72_c7f9_43a9_83ce_40ab705d092a.slice/crio-32b0a67fa4b076d087a33522b203440d41d361c669fa0002eac4b8bd62ec9de3 WatchSource:0}: Error finding container 32b0a67fa4b076d087a33522b203440d41d361c669fa0002eac4b8bd62ec9de3: Status 404 returned error can't find the container with id 32b0a67fa4b076d087a33522b203440d41d361c669fa0002eac4b8bd62ec9de3
Nov 01 06:11:51.014151 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:51.014123    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 06:11:51.014237 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:51.014183    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 06:11:51.014237 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:51.014210    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 06:11:51.706270 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:51.706229    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cd9pn" event={"ID":"ce6f3d72-c7f9-43a9-83ce-40ab705d092a","Type":"ContainerStarted","Data":"bdfdb3c191bba3523c1d0ad9d093a267e231d9842a6ab998504fd9db228be467"}
Nov 01 06:11:51.706718 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:51.706280    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cd9pn" event={"ID":"ce6f3d72-c7f9-43a9-83ce-40ab705d092a","Type":"ContainerStarted","Data":"32b0a67fa4b076d087a33522b203440d41d361c669fa0002eac4b8bd62ec9de3"}
Nov 01 06:11:52.593485 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:52.591283    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cd9pn"
Nov 01 06:11:52.593485 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:52.591430    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-cd9pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Nov 01 06:11:52.593485 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:52.591504    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cd9pn" podUID="ce6f3d72-c7f9-43a9-83ce-40ab705d092a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Nov 01 06:11:53.591483 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:53.590817    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-cd9pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Nov 01 06:11:53.591483 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:53.590892    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cd9pn" podUID="ce6f3d72-c7f9-43a9-83ce-40ab705d092a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Nov 01 06:11:54.595477 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:54.591778    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-cd9pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Nov 01 06:11:54.595477 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:54.591848    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cd9pn" podUID="ce6f3d72-c7f9-43a9-83ce-40ab705d092a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Nov 01 06:11:55.587824 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:55.587786    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-cd9pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Nov 01 06:11:55.588103 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:55.588074    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cd9pn" podUID="ce6f3d72-c7f9-43a9-83ce-40ab705d092a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Nov 01 06:11:56.591862 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:56.591824    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-cd9pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Nov 01 06:11:56.592474 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:56.592414    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cd9pn" podUID="ce6f3d72-c7f9-43a9-83ce-40ab705d092a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Nov 01 06:11:57.582999 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:57.582963    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-cd9pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Nov 01 06:11:57.583168 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:57.583026    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cd9pn" podUID="ce6f3d72-c7f9-43a9-83ce-40ab705d092a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Nov 01 06:11:57.701546 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:57.701494    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cd9pn" podStartSLOduration=7.701478106 podStartE2EDuration="7.701478106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:11:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:11:51.889661742 +0000 UTC m=+4089.960458464" watchObservedRunningTime="2026-11-01 06:11:57.701478106 +0000 UTC m=+4095.772274819"
Nov 01 06:11:57.702010 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:57.701995    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:11:59.582711 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:59.582646    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-cd9pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 06:11:59.583408 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:11:59.583363    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cd9pn" podUID="ce6f3d72-c7f9-43a9-83ce-40ab705d092a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 06:12:00.069030 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:12:00.068996    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 06:12:00.624645 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:12:00.624601    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-cd9pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 06:12:00.625133 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:12:00.624677    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cd9pn" podUID="ce6f3d72-c7f9-43a9-83ce-40ab705d092a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 06:12:01.192425 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:12:01.192397    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cd9pn"
Nov 01 06:12:01.192715 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:12:01.192697    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cd9pn"
Nov 01 06:12:01.193410 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:12:01.193389    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cd9pn"
Nov 01 06:12:05.380851 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:12:05.380821    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tgfm2"]
Nov 01 06:12:05.384874 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:12:05.384849    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tgfm2"
Nov 01 06:12:05.469193 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:12:05.463768    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5665e928-b4df-455f-99c1-d466700fee62-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-tgfm2\" (UID: \"5665e928-b4df-455f-99c1-d466700fee62\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tgfm2"
Nov 01 06:12:05.469193 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:12:05.463817    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5665e928-b4df-455f-99c1-d466700fee62-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-tgfm2\" (UID: \"5665e928-b4df-455f-99c1-d466700fee62\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tgfm2"
Nov 01 06:12:05.469193 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:12:05.463857    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l29m4\" (UniqueName: \"kubernetes.io/projected/5665e928-b4df-455f-99c1-d466700fee62-kube-api-access-l29m4\") pod \"maas-default-gateway-openshift-default-58d898bd55-tgfm2\" (UID: \"5665e928-b4df-455f-99c1-d466700fee62\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tgfm2"
Nov 01 06:12:05.469193 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:12:05.463891    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5665e928-b4df-455f-99c1-d466700fee62-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-tgfm2\" (UID: \"5665e928-b4df-455f-99c1-d466700fee62\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tgfm2"
Nov 01 06:12:05.469193 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:12:05.463944    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5665e928-b4df-455f-99c1-d466700fee62-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-tgfm2\" (UID: \"5665e928-b4df-455f-99c1-d466700fee62\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tgfm2"
Nov 01 06:12:05.469193 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:12:05.464003    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5665e928-b4df-455f-99c1-d466700fee62-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-tgfm2\" (UID: \"5665e928-b4df-455f-99c1-d466700fee62\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tgfm2"
Nov 01 06:12:05.469193 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:12:05.464045    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5665e928-b4df-455f-99c1-d466700fee62-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-tgfm2\" (UID: \"5665e928-b4df-455f-99c1-d466700fee62\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tgfm2"
Nov 01 06:12:05.469193 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:12:05.464083    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5665e928-b4df-455f-99c1-d466700fee62-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-tgfm2\" (UID: \"5665e928-b4df-455f-99c1-d466700fee62\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tgfm2"
Nov 01 06:12:05.469193 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:12:05.464167    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5665e928-b4df-455f-99c1-d466700fee62-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-tgfm2\" (UID: \"5665e928-b4df-455f-99c1-d466700fee62\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tgfm2"
Nov 01 06:12:05.482644 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:12:05.482614    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tgfm2"]
Nov 01 06:12:05.565253 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:12:05.565218    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5665e928-b4df-455f-99c1-d466700fee62-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-tgfm2\" (UID: \"5665e928-b4df-455f-99c1-d466700fee62\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tgfm2"
Nov 01 06:12:05.565253 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:12:05.565255    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5665e928-b4df-455f-99c1-d466700fee62-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-tgfm2\" (UID: \"5665e928-b4df-455f-99c1-d466700fee62\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tgfm2"
Nov 01 06:12:05.565559 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:12:05.565273    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5665e928-b4df-455f-99c1-d466700fee62-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-tgfm2\" (UID: \"5665e928-b4df-455f-99c1-d466700fee62\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tgfm2"
Nov 01 06:12:05.565559 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:12:05.565308    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5665e928-b4df-455f-99c1-d466700fee62-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-tgfm2\" (UID: \"5665e928-b4df-455f-99c1-d466700fee62\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tgfm2"
Nov 01 06:12:05.565559 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:12:05.565343    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5665e928-b4df-455f-99c1-d466700fee62-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-tgfm2\" (UID: \"5665e928-b4df-455f-99c1-d466700fee62\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tgfm2"
Nov 01 06:12:05.565559 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:12:05.565358    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5665e928-b4df-455f-99c1-d466700fee62-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-tgfm2\" (UID: \"5665e928-b4df-455f-99c1-d466700fee62\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tgfm2"
Nov 01 06:12:05.565559 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:12:05.565376    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l29m4\" (UniqueName: \"kubernetes.io/projected/5665e928-b4df-455f-99c1-d466700fee62-kube-api-access-l29m4\") pod \"maas-default-gateway-openshift-default-58d898bd55-tgfm2\" (UID: \"5665e928-b4df-455f-99c1-d466700fee62\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tgfm2"
Nov 01 06:12:05.565559 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:12:05.565403    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5665e928-b4df-455f-99c1-d466700fee62-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-tgfm2\" (UID: \"5665e928-b4df-455f-99c1-d466700fee62\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tgfm2"
Nov 01 06:12:05.565559 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:12:05.565478    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5665e928-b4df-455f-99c1-d466700fee62-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-tgfm2\" (UID: \"5665e928-b4df-455f-99c1-d466700fee62\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tgfm2"
Nov 01 06:12:05.565945 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:12:05.565722    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5665e928-b4df-455f-99c1-d466700fee62-workload-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-tgfm2\" (UID: \"5665e928-b4df-455f-99c1-d466700fee62\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tgfm2"
Nov 01 06:12:05.565945 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:12:05.565816    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5665e928-b4df-455f-99c1-d466700fee62-workload-certs\") pod \"maas-default-gateway-openshift-default-58d898bd55-tgfm2\" (UID: \"5665e928-b4df-455f-99c1-d466700fee62\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tgfm2"
Nov 01 06:12:05.566055 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:12:05.565965    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5665e928-b4df-455f-99c1-d466700fee62-credential-socket\") pod \"maas-default-gateway-openshift-default-58d898bd55-tgfm2\" (UID: \"5665e928-b4df-455f-99c1-d466700fee62\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tgfm2"
Nov 01 06:12:05.566055 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:12:05.566001    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5665e928-b4df-455f-99c1-d466700fee62-istio-data\") pod \"maas-default-gateway-openshift-default-58d898bd55-tgfm2\" (UID: \"5665e928-b4df-455f-99c1-d466700fee62\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tgfm2"
Nov 01 06:12:05.566336 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:12:05.566196    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5665e928-b4df-455f-99c1-d466700fee62-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58d898bd55-tgfm2\" (UID: \"5665e928-b4df-455f-99c1-d466700fee62\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tgfm2"
Nov 01 06:12:05.567774 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:12:05.567752    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5665e928-b4df-455f-99c1-d466700fee62-istio-envoy\") pod \"maas-default-gateway-openshift-default-58d898bd55-tgfm2\" (UID: \"5665e928-b4df-455f-99c1-d466700fee62\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tgfm2"
Nov 01 06:12:05.567892 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:12:05.567876    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5665e928-b4df-455f-99c1-d466700fee62-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58d898bd55-tgfm2\" (UID: \"5665e928-b4df-455f-99c1-d466700fee62\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tgfm2"
Nov 01 06:12:05.625679 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:12:05.625652    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5665e928-b4df-455f-99c1-d466700fee62-istio-token\") pod \"maas-default-gateway-openshift-default-58d898bd55-tgfm2\" (UID: \"5665e928-b4df-455f-99c1-d466700fee62\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tgfm2"
Nov 01 06:12:05.625906 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:12:05.625889    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l29m4\" (UniqueName: \"kubernetes.io/projected/5665e928-b4df-455f-99c1-d466700fee62-kube-api-access-l29m4\") pod \"maas-default-gateway-openshift-default-58d898bd55-tgfm2\" (UID: \"5665e928-b4df-455f-99c1-d466700fee62\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tgfm2"
Nov 01 06:12:05.698312 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:12:05.698279    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tgfm2"
Nov 01 06:12:06.126488 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:12:06.126392    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tgfm2"]
Nov 01 06:12:06.129774 ip-10-0-129-92 kubenswrapper[5689]: W1101 06:12:06.129742    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5665e928_b4df_455f_99c1_d466700fee62.slice/crio-d887736764d8b74936dd92c7f8c41d66fa57bd4a0daddbee840e0b66658c7e27 WatchSource:0}: Error finding container d887736764d8b74936dd92c7f8c41d66fa57bd4a0daddbee840e0b66658c7e27: Status 404 returned error can't find the container with id d887736764d8b74936dd92c7f8c41d66fa57bd4a0daddbee840e0b66658c7e27
Nov 01 06:12:06.131899 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:12:06.131865    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 06:12:06.131990 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:12:06.131930    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 06:12:06.131990 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:12:06.131956    5689 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 01 06:12:06.787473 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:12:06.787423    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tgfm2" event={"ID":"5665e928-b4df-455f-99c1-d466700fee62","Type":"ContainerStarted","Data":"805587c8ed464458cce11799ebe0137ce2c5699fbeae7fe26d44b90a1bbc7e92"}
Nov 01 06:12:06.788266 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:12:06.788144    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tgfm2" event={"ID":"5665e928-b4df-455f-99c1-d466700fee62","Type":"ContainerStarted","Data":"d887736764d8b74936dd92c7f8c41d66fa57bd4a0daddbee840e0b66658c7e27"}
Nov 01 06:12:06.909938 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:12:06.909883    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tgfm2" podStartSLOduration=1.909867487 podStartE2EDuration="1.909867487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:12:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:12:06.908266892 +0000 UTC m=+4104.979063834" watchObservedRunningTime="2026-11-01 06:12:06.909867487 +0000 UTC m=+4104.980664206"
Nov 01 06:12:07.401660 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:12:07.401618    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:12:07.699751 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:12:07.699663    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tgfm2"
Nov 01 06:12:07.701202 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:12:07.701174    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-tgfm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Nov 01 06:12:07.701334 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:12:07.701222    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tgfm2" podUID="5665e928-b4df-455f-99c1-d466700fee62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Nov 01 06:12:08.699849 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:12:08.699812    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-tgfm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Nov 01 06:12:08.700241 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:12:08.699872    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tgfm2" podUID="5665e928-b4df-455f-99c1-d466700fee62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Nov 01 06:12:09.698852 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:12:09.698817    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-tgfm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Nov 01 06:12:09.699147 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:12:09.699115    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tgfm2" podUID="5665e928-b4df-455f-99c1-d466700fee62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Nov 01 06:12:10.699777 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:12:10.699742    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-tgfm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Nov 01 06:12:10.700335 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:12:10.699793    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tgfm2" podUID="5665e928-b4df-455f-99c1-d466700fee62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Nov 01 06:12:11.699858 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:12:11.699819    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-tgfm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Nov 01 06:12:11.700290 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:12:11.699885    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tgfm2" podUID="5665e928-b4df-455f-99c1-d466700fee62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Nov 01 06:12:12.698831 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:12:12.698789    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-tgfm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Nov 01 06:12:12.699047 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:12:12.698858    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tgfm2" podUID="5665e928-b4df-455f-99c1-d466700fee62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Nov 01 06:12:14.749484 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:12:14.747541    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-tgfm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 06:12:14.749484 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:12:14.747621    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tgfm2" podUID="5665e928-b4df-455f-99c1-d466700fee62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 06:12:15.748713 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:12:15.748678    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-tgfm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 06:12:15.748936 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:12:15.748735    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tgfm2" podUID="5665e928-b4df-455f-99c1-d466700fee62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 06:12:16.789671 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:12:16.789623    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-tgfm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 06:12:16.790181 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:12:16.789708    5689 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tgfm2" podUID="5665e928-b4df-455f-99c1-d466700fee62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 06:12:16.976261 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:12:16.976212    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:12:17.174169 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:12:17.174142    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:12:17.228565 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:12:17.228531    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tgfm2"
Nov 01 06:12:17.229021 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:12:17.229001    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tgfm2"
Nov 01 06:12:17.229785 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:12:17.229764    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tgfm2"
Nov 01 06:12:25.704667 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:12:25.704631    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:12:30.068265 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:12:30.068222    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 06:12:44.503086 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:12:44.503042    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:13:00.070965 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:13:00.070874    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 06:13:03.099079 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:13:03.099045    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:13:05.548980 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:13:05.548949    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:13:06.145948 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:13:06.145916    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-25304531-kserve-router-scheduler-5664d6b58f-jvsk9"]
Nov 01 06:13:06.149577 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:13:06.149557    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-25304531-kserve-router-scheduler-5664d6b58f-jvsk9"
Nov 01 06:13:06.228403 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:13:06.228370    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-25304531-epp-sa-dockercfg-pg9h9\""
Nov 01 06:13:06.228694 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:13:06.228673    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-25304531-kserve-self-signed-certs\""
Nov 01 06:13:06.246280 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:13:06.246244    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkvqt\" (UniqueName: \"kubernetes.io/projected/258ac054-4818-4155-b3f2-c9fe1aae097f-kube-api-access-gkvqt\") pod \"e2e-prio-pool-25304531-kserve-router-scheduler-5664d6b58f-jvsk9\" (UID: \"258ac054-4818-4155-b3f2-c9fe1aae097f\") " pod="llm/e2e-prio-pool-25304531-kserve-router-scheduler-5664d6b58f-jvsk9"
Nov 01 06:13:06.246409 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:13:06.246317    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/258ac054-4818-4155-b3f2-c9fe1aae097f-tls-certs\") pod \"e2e-prio-pool-25304531-kserve-router-scheduler-5664d6b58f-jvsk9\" (UID: \"258ac054-4818-4155-b3f2-c9fe1aae097f\") " pod="llm/e2e-prio-pool-25304531-kserve-router-scheduler-5664d6b58f-jvsk9"
Nov 01 06:13:06.252494 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:13:06.252476    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-25304531-kserve-router-scheduler-5664d6b58f-jvsk9"]
Nov 01 06:13:06.276414 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:13:06.276386    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:13:06.347749 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:13:06.347716    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkvqt\" (UniqueName: \"kubernetes.io/projected/258ac054-4818-4155-b3f2-c9fe1aae097f-kube-api-access-gkvqt\") pod \"e2e-prio-pool-25304531-kserve-router-scheduler-5664d6b58f-jvsk9\" (UID: \"258ac054-4818-4155-b3f2-c9fe1aae097f\") " pod="llm/e2e-prio-pool-25304531-kserve-router-scheduler-5664d6b58f-jvsk9"
Nov 01 06:13:06.347911 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:13:06.347762    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/258ac054-4818-4155-b3f2-c9fe1aae097f-tls-certs\") pod \"e2e-prio-pool-25304531-kserve-router-scheduler-5664d6b58f-jvsk9\" (UID: \"258ac054-4818-4155-b3f2-c9fe1aae097f\") " pod="llm/e2e-prio-pool-25304531-kserve-router-scheduler-5664d6b58f-jvsk9"
Nov 01 06:13:06.350399 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:13:06.350374    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/258ac054-4818-4155-b3f2-c9fe1aae097f-tls-certs\") pod \"e2e-prio-pool-25304531-kserve-router-scheduler-5664d6b58f-jvsk9\" (UID: \"258ac054-4818-4155-b3f2-c9fe1aae097f\") " pod="llm/e2e-prio-pool-25304531-kserve-router-scheduler-5664d6b58f-jvsk9"
Nov 01 06:13:06.408378 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:13:06.408352    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkvqt\" (UniqueName: \"kubernetes.io/projected/258ac054-4818-4155-b3f2-c9fe1aae097f-kube-api-access-gkvqt\") pod \"e2e-prio-pool-25304531-kserve-router-scheduler-5664d6b58f-jvsk9\" (UID: \"258ac054-4818-4155-b3f2-c9fe1aae097f\") " pod="llm/e2e-prio-pool-25304531-kserve-router-scheduler-5664d6b58f-jvsk9"
Nov 01 06:13:06.461844 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:13:06.461813    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-25304531-kserve-router-scheduler-5664d6b58f-jvsk9"
Nov 01 06:13:06.837029 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:13:06.835043    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-25304531-kserve-router-scheduler-5664d6b58f-jvsk9"]
Nov 01 06:13:06.844124 ip-10-0-129-92 kubenswrapper[5689]: W1101 06:13:06.844087    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod258ac054_4818_4155_b3f2_c9fe1aae097f.slice/crio-c65a1252a0217165bc1b9315ef93e0664e420c9b71af6b7894c09d8883687925 WatchSource:0}: Error finding container c65a1252a0217165bc1b9315ef93e0664e420c9b71af6b7894c09d8883687925: Status 404 returned error can't find the container with id c65a1252a0217165bc1b9315ef93e0664e420c9b71af6b7894c09d8883687925
Nov 01 06:13:07.153661 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:13:07.153628    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-25304531-kserve-router-scheduler-5664d6b58f-jvsk9" event={"ID":"258ac054-4818-4155-b3f2-c9fe1aae097f","Type":"ContainerStarted","Data":"c65a1252a0217165bc1b9315ef93e0664e420c9b71af6b7894c09d8883687925"}
Nov 01 06:13:09.165138 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:13:09.165105    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-25304531-kserve-router-scheduler-5664d6b58f-jvsk9" event={"ID":"258ac054-4818-4155-b3f2-c9fe1aae097f","Type":"ContainerStarted","Data":"20a1a71bf9b06494c07c7714e93cd21c9970803df3adf460c3cc1b0301be38f2"}
Nov 01 06:13:09.165591 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:13:09.165342    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-25304531-kserve-router-scheduler-5664d6b58f-jvsk9"
Nov 01 06:13:09.282521 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:13:09.282475    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-25304531-kserve-router-scheduler-5664d6b58f-jvsk9" podStartSLOduration=1.3657371600000001 podStartE2EDuration="3.282443415s" totalImagesPullingTime="1.916706255s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:13:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:13:09.280413991 +0000 UTC m=+4167.351210704" watchObservedRunningTime="2026-11-01 06:13:09.282443415 +0000 UTC m=+4167.353240134"
Nov 01 06:13:16.792343 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:13:16.792307    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:13:17.485325 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:13:17.485295    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:13:25.624276 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:13:25.624243    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:13:30.068510 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:13:30.068471    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 06:13:40.171567 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:13:40.171537    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-25304531-kserve-router-scheduler-5664d6b58f-jvsk9"
Nov 01 06:13:48.899379 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:13:48.899329    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:14:00.068804 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:14:00.068769    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 06:14:12.388030 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:14:12.387988    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:14:12.872266 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:14:12.872239    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:14:12.957406 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:14:12.957373    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-25304531-kserve-router-scheduler-5664d6b58f-jvsk9"]
Nov 01 06:14:14.270398 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:14:14.270364    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:14:17.828160 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:14:17.828120    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:14:20.213537 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:14:20.203537    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-25304531-kserve-router-scheduler-5664d6b58f-jvsk9" podUID="258ac054-4818-4155-b3f2-c9fe1aae097f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:14:23.977948 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:14:23.977911    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:14:27.977971 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:14:27.977928    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-25304531-kserve-router-scheduler-5664d6b58f-jvsk9" podUID="258ac054-4818-4155-b3f2-c9fe1aae097f" containerName="main" containerID="cri-o://20a1a71bf9b06494c07c7714e93cd21c9970803df3adf460c3cc1b0301be38f2" gracePeriod=45
Nov 01 06:14:30.068542 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:14:30.068507    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 06:14:30.171598 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:14:30.171555    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-25304531-kserve-router-scheduler-5664d6b58f-jvsk9" podUID="258ac054-4818-4155-b3f2-c9fe1aae097f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:14:39.841587 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:14:39.841556    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:14:40.171293 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:14:40.171263    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-25304531-kserve-router-scheduler-5664d6b58f-jvsk9" podUID="258ac054-4818-4155-b3f2-c9fe1aae097f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:14:40.171478 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:14:40.171362    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-25304531-kserve-router-scheduler-5664d6b58f-jvsk9"
Nov 01 06:14:44.212951 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:14:44.212913    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:14:47.198486 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:14:47.197556    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-w2nmc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 06:14:47.198486 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:14:47.197641    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-w2nmc" podUID="e6319ad1-4be9-44bb-898a-139e04529f67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 06:14:47.239587 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:14:47.239553    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-cd9pn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 06:14:47.239833 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:14:47.239805    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cd9pn" podUID="ce6f3d72-c7f9-43a9-83ce-40ab705d092a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 06:14:47.265659 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:14:47.265542    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-c6zlb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 06:14:47.265659 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:14:47.265597    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c6zlb" podUID="d1d4ddb1-fd17-4ac1-ada4-f785008a0913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 06:14:50.172179 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:14:50.172138    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-25304531-kserve-router-scheduler-5664d6b58f-jvsk9" podUID="258ac054-4818-4155-b3f2-c9fe1aae097f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:14:58.224741 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:14:58.224715    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-25304531-kserve-router-scheduler-5664d6b58f-jvsk9"
Nov 01 06:14:58.314938 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:14:58.314843    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/258ac054-4818-4155-b3f2-c9fe1aae097f-kube-api-access-gkvqt\" (UniqueName: \"kubernetes.io/projected/258ac054-4818-4155-b3f2-c9fe1aae097f-kube-api-access-gkvqt\") pod \"258ac054-4818-4155-b3f2-c9fe1aae097f\" (UID: \"258ac054-4818-4155-b3f2-c9fe1aae097f\") "
Nov 01 06:14:58.314938 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:14:58.314888    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/258ac054-4818-4155-b3f2-c9fe1aae097f-tls-certs\" (UniqueName: \"kubernetes.io/secret/258ac054-4818-4155-b3f2-c9fe1aae097f-tls-certs\") pod \"258ac054-4818-4155-b3f2-c9fe1aae097f\" (UID: \"258ac054-4818-4155-b3f2-c9fe1aae097f\") "
Nov 01 06:14:58.317182 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:14:58.317148    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/258ac054-4818-4155-b3f2-c9fe1aae097f-tls-certs" pod "258ac054-4818-4155-b3f2-c9fe1aae097f" (UID: "258ac054-4818-4155-b3f2-c9fe1aae097f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:14:58.317312 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:14:58.317209    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/258ac054-4818-4155-b3f2-c9fe1aae097f-kube-api-access-gkvqt" pod "258ac054-4818-4155-b3f2-c9fe1aae097f" (UID: "258ac054-4818-4155-b3f2-c9fe1aae097f"). InnerVolumeSpecName "kube-api-access-gkvqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:14:58.415842 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:14:58.415811    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkvqt\" (UniqueName: \"kubernetes.io/projected/258ac054-4818-4155-b3f2-c9fe1aae097f-kube-api-access-gkvqt\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:14:58.415842 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:14:58.415837    5689 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/258ac054-4818-4155-b3f2-c9fe1aae097f-tls-certs\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:14:58.796765 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:14:58.796731    5689 generic.go:356] "Generic (PLEG): container finished" podID="258ac054-4818-4155-b3f2-c9fe1aae097f" containerID="20a1a71bf9b06494c07c7714e93cd21c9970803df3adf460c3cc1b0301be38f2" exitCode=0
Nov 01 06:14:58.796959 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:14:58.796779    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-25304531-kserve-router-scheduler-5664d6b58f-jvsk9" event={"ID":"258ac054-4818-4155-b3f2-c9fe1aae097f","Type":"ContainerDied","Data":"20a1a71bf9b06494c07c7714e93cd21c9970803df3adf460c3cc1b0301be38f2"}
Nov 01 06:14:58.796959 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:14:58.796820    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-25304531-kserve-router-scheduler-5664d6b58f-jvsk9" event={"ID":"258ac054-4818-4155-b3f2-c9fe1aae097f","Type":"ContainerDied","Data":"c65a1252a0217165bc1b9315ef93e0664e420c9b71af6b7894c09d8883687925"}
Nov 01 06:14:58.796959 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:14:58.796840    5689 scope.go:122] "RemoveContainer" containerID="20a1a71bf9b06494c07c7714e93cd21c9970803df3adf460c3cc1b0301be38f2"
Nov 01 06:14:58.796959 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:14:58.796840    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-25304531-kserve-router-scheduler-5664d6b58f-jvsk9"
Nov 01 06:14:58.805841 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:14:58.805825    5689 scope.go:122] "RemoveContainer" containerID="20a1a71bf9b06494c07c7714e93cd21c9970803df3adf460c3cc1b0301be38f2"
Nov 01 06:14:58.806104 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:14:58.806085    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20a1a71bf9b06494c07c7714e93cd21c9970803df3adf460c3cc1b0301be38f2\": container with ID starting with 20a1a71bf9b06494c07c7714e93cd21c9970803df3adf460c3cc1b0301be38f2 not found: ID does not exist" containerID="20a1a71bf9b06494c07c7714e93cd21c9970803df3adf460c3cc1b0301be38f2"
Nov 01 06:14:58.806180 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:14:58.806112    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20a1a71bf9b06494c07c7714e93cd21c9970803df3adf460c3cc1b0301be38f2"} err="failed to get container status \"20a1a71bf9b06494c07c7714e93cd21c9970803df3adf460c3cc1b0301be38f2\": rpc error: code = NotFound desc = could not find container \"20a1a71bf9b06494c07c7714e93cd21c9970803df3adf460c3cc1b0301be38f2\": container with ID starting with 20a1a71bf9b06494c07c7714e93cd21c9970803df3adf460c3cc1b0301be38f2 not found: ID does not exist"
Nov 01 06:14:58.958829 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:14:58.958804    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-25304531-kserve-router-scheduler-5664d6b58f-jvsk9"]
Nov 01 06:14:59.007750 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:14:59.007726    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-25304531-kserve-router-scheduler-5664d6b58f-jvsk9"]
Nov 01 06:15:00.067123 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:15:00.067095    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="258ac054-4818-4155-b3f2-c9fe1aae097f" path="/var/lib/kubelet/pods/258ac054-4818-4155-b3f2-c9fe1aae097f/volumes"
Nov 01 06:15:00.068308 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:15:00.068278    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 06:15:06.182466 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:15:06.182431    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-f6cbc132-kserve-6948cd4f9c-zlqx8"]
Nov 01 06:15:06.182923 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:15:06.182840    5689 cpu_manager.go:414] "containerMap: removing container" podUID="258ac054-4818-4155-b3f2-c9fe1aae097f" containerName="main"
Nov 01 06:15:06.182923 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:15:06.182852    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="258ac054-4818-4155-b3f2-c9fe1aae097f" containerName="main"
Nov 01 06:15:06.182923 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:15:06.182908    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="258ac054-4818-4155-b3f2-c9fe1aae097f" containerName="main"
Nov 01 06:15:06.187094 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:15:06.187078    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-f6cbc132-kserve-6948cd4f9c-zlqx8"
Nov 01 06:15:06.288633 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:15:06.288597    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-f6cbc132-kserve-self-signed-certs\""
Nov 01 06:15:06.290065 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:15:06.290045    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-f6cbc132-kserve-6948cd4f9c-zlqx8"]
Nov 01 06:15:06.292025 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:15:06.292006    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2276668-6bb8-4a7b-8502-d83ff5667027-dshm\") pod \"e2e-same-id-b-f6cbc132-kserve-6948cd4f9c-zlqx8\" (UID: \"f2276668-6bb8-4a7b-8502-d83ff5667027\") " pod="llm/e2e-same-id-b-f6cbc132-kserve-6948cd4f9c-zlqx8"
Nov 01 06:15:06.292072 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:15:06.292049    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2276668-6bb8-4a7b-8502-d83ff5667027-model-cache\") pod \"e2e-same-id-b-f6cbc132-kserve-6948cd4f9c-zlqx8\" (UID: \"f2276668-6bb8-4a7b-8502-d83ff5667027\") " pod="llm/e2e-same-id-b-f6cbc132-kserve-6948cd4f9c-zlqx8"
Nov 01 06:15:06.292183 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:15:06.292168    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlxr8\" (UniqueName: \"kubernetes.io/projected/f2276668-6bb8-4a7b-8502-d83ff5667027-kube-api-access-vlxr8\") pod \"e2e-same-id-b-f6cbc132-kserve-6948cd4f9c-zlqx8\" (UID: \"f2276668-6bb8-4a7b-8502-d83ff5667027\") " pod="llm/e2e-same-id-b-f6cbc132-kserve-6948cd4f9c-zlqx8"
Nov 01 06:15:06.292243 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:15:06.292231    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2276668-6bb8-4a7b-8502-d83ff5667027-home\") pod \"e2e-same-id-b-f6cbc132-kserve-6948cd4f9c-zlqx8\" (UID: \"f2276668-6bb8-4a7b-8502-d83ff5667027\") " pod="llm/e2e-same-id-b-f6cbc132-kserve-6948cd4f9c-zlqx8"
Nov 01 06:15:06.292282 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:15:06.292254    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2276668-6bb8-4a7b-8502-d83ff5667027-tls-certs\") pod \"e2e-same-id-b-f6cbc132-kserve-6948cd4f9c-zlqx8\" (UID: \"f2276668-6bb8-4a7b-8502-d83ff5667027\") " pod="llm/e2e-same-id-b-f6cbc132-kserve-6948cd4f9c-zlqx8"
Nov 01 06:15:06.292371 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:15:06.292342    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2276668-6bb8-4a7b-8502-d83ff5667027-tmp-dir\") pod \"e2e-same-id-b-f6cbc132-kserve-6948cd4f9c-zlqx8\" (UID: \"f2276668-6bb8-4a7b-8502-d83ff5667027\") " pod="llm/e2e-same-id-b-f6cbc132-kserve-6948cd4f9c-zlqx8"
Nov 01 06:15:06.317712 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:15:06.317686    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:15:06.393447 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:15:06.393412    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2276668-6bb8-4a7b-8502-d83ff5667027-dshm\") pod \"e2e-same-id-b-f6cbc132-kserve-6948cd4f9c-zlqx8\" (UID: \"f2276668-6bb8-4a7b-8502-d83ff5667027\") " pod="llm/e2e-same-id-b-f6cbc132-kserve-6948cd4f9c-zlqx8"
Nov 01 06:15:06.393693 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:15:06.393524    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2276668-6bb8-4a7b-8502-d83ff5667027-model-cache\") pod \"e2e-same-id-b-f6cbc132-kserve-6948cd4f9c-zlqx8\" (UID: \"f2276668-6bb8-4a7b-8502-d83ff5667027\") " pod="llm/e2e-same-id-b-f6cbc132-kserve-6948cd4f9c-zlqx8"
Nov 01 06:15:06.393693 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:15:06.393584    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlxr8\" (UniqueName: \"kubernetes.io/projected/f2276668-6bb8-4a7b-8502-d83ff5667027-kube-api-access-vlxr8\") pod \"e2e-same-id-b-f6cbc132-kserve-6948cd4f9c-zlqx8\" (UID: \"f2276668-6bb8-4a7b-8502-d83ff5667027\") " pod="llm/e2e-same-id-b-f6cbc132-kserve-6948cd4f9c-zlqx8"
Nov 01 06:15:06.393693 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:15:06.393615    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2276668-6bb8-4a7b-8502-d83ff5667027-home\") pod \"e2e-same-id-b-f6cbc132-kserve-6948cd4f9c-zlqx8\" (UID: \"f2276668-6bb8-4a7b-8502-d83ff5667027\") " pod="llm/e2e-same-id-b-f6cbc132-kserve-6948cd4f9c-zlqx8"
Nov 01 06:15:06.393693 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:15:06.393641    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2276668-6bb8-4a7b-8502-d83ff5667027-tls-certs\") pod \"e2e-same-id-b-f6cbc132-kserve-6948cd4f9c-zlqx8\" (UID: \"f2276668-6bb8-4a7b-8502-d83ff5667027\") " pod="llm/e2e-same-id-b-f6cbc132-kserve-6948cd4f9c-zlqx8"
Nov 01 06:15:06.393951 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:15:06.393723    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2276668-6bb8-4a7b-8502-d83ff5667027-tmp-dir\") pod \"e2e-same-id-b-f6cbc132-kserve-6948cd4f9c-zlqx8\" (UID: \"f2276668-6bb8-4a7b-8502-d83ff5667027\") " pod="llm/e2e-same-id-b-f6cbc132-kserve-6948cd4f9c-zlqx8"
Nov 01 06:15:06.394053 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:15:06.394030    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2276668-6bb8-4a7b-8502-d83ff5667027-model-cache\") pod \"e2e-same-id-b-f6cbc132-kserve-6948cd4f9c-zlqx8\" (UID: \"f2276668-6bb8-4a7b-8502-d83ff5667027\") " pod="llm/e2e-same-id-b-f6cbc132-kserve-6948cd4f9c-zlqx8"
Nov 01 06:15:06.394113 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:15:06.394065    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2276668-6bb8-4a7b-8502-d83ff5667027-home\") pod \"e2e-same-id-b-f6cbc132-kserve-6948cd4f9c-zlqx8\" (UID: \"f2276668-6bb8-4a7b-8502-d83ff5667027\") " pod="llm/e2e-same-id-b-f6cbc132-kserve-6948cd4f9c-zlqx8"
Nov 01 06:15:06.394113 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:15:06.394082    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2276668-6bb8-4a7b-8502-d83ff5667027-tmp-dir\") pod \"e2e-same-id-b-f6cbc132-kserve-6948cd4f9c-zlqx8\" (UID: \"f2276668-6bb8-4a7b-8502-d83ff5667027\") " pod="llm/e2e-same-id-b-f6cbc132-kserve-6948cd4f9c-zlqx8"
Nov 01 06:15:06.395885 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:15:06.395860    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2276668-6bb8-4a7b-8502-d83ff5667027-dshm\") pod \"e2e-same-id-b-f6cbc132-kserve-6948cd4f9c-zlqx8\" (UID: \"f2276668-6bb8-4a7b-8502-d83ff5667027\") " pod="llm/e2e-same-id-b-f6cbc132-kserve-6948cd4f9c-zlqx8"
Nov 01 06:15:06.396223 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:15:06.396203    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2276668-6bb8-4a7b-8502-d83ff5667027-tls-certs\") pod \"e2e-same-id-b-f6cbc132-kserve-6948cd4f9c-zlqx8\" (UID: \"f2276668-6bb8-4a7b-8502-d83ff5667027\") " pod="llm/e2e-same-id-b-f6cbc132-kserve-6948cd4f9c-zlqx8"
Nov 01 06:15:06.454443 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:15:06.454366    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlxr8\" (UniqueName: \"kubernetes.io/projected/f2276668-6bb8-4a7b-8502-d83ff5667027-kube-api-access-vlxr8\") pod \"e2e-same-id-b-f6cbc132-kserve-6948cd4f9c-zlqx8\" (UID: \"f2276668-6bb8-4a7b-8502-d83ff5667027\") " pod="llm/e2e-same-id-b-f6cbc132-kserve-6948cd4f9c-zlqx8"
Nov 01 06:15:06.507703 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:15:06.507672    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-f6cbc132-kserve-6948cd4f9c-zlqx8"
Nov 01 06:15:06.910148 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:15:06.910116    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-f6cbc132-kserve-6948cd4f9c-zlqx8"]
Nov 01 06:15:07.845433 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:15:07.845398    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-f6cbc132-kserve-6948cd4f9c-zlqx8" event={"ID":"f2276668-6bb8-4a7b-8502-d83ff5667027","Type":"ContainerStarted","Data":"18588daff0988ea4c966869277b3b48516badc635999f1d225fe6e1994a4091d"}
Nov 01 06:15:07.845883 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:15:07.845440    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-f6cbc132-kserve-6948cd4f9c-zlqx8" event={"ID":"f2276668-6bb8-4a7b-8502-d83ff5667027","Type":"ContainerStarted","Data":"6a1989aa7717a910f98d3fe020e6c9a01ff63df7d734decd8a7c171d9d21a7a1"}
Nov 01 06:15:07.962359 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:15:07.962305    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-f6cbc132-kserve-6948cd4f9c-zlqx8" podStartSLOduration=1.962291395 podStartE2EDuration="1.962291395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:15:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:15:07.960723816 +0000 UTC m=+4286.031520536" watchObservedRunningTime="2026-11-01 06:15:07.962291395 +0000 UTC m=+4286.033088115"
Nov 01 06:15:08.508538 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:15:08.508506    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-f6cbc132-kserve-6948cd4f9c-zlqx8"
Nov 01 06:15:10.907176 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:15:10.907146    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:15:16.508013 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:15:16.507979    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-f6cbc132-kserve-6948cd4f9c-zlqx8"
Nov 01 06:15:16.520346 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:15:16.520323    5689 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-f6cbc132-kserve-6948cd4f9c-zlqx8"
Nov 01 06:15:17.242823 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:15:17.242796    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-f6cbc132-kserve-6948cd4f9c-zlqx8"
Nov 01 06:15:18.178028 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:15:18.177994    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:15:30.068097 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:15:30.068059    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 06:15:36.066088 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:15:36.066052    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:15:47.612864 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:15:47.612831    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:15:51.296394 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:15:51.296316    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:15:56.094249 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:15:56.094219    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-f6cbc132-kserve-6948cd4f9c-zlqx8"]
Nov 01 06:15:56.136223 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:15:56.136194    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:16:00.069925 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:16:00.069891    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 06:16:02.199478 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:16:02.198563    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-w2nmc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 06:16:02.199478 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:16:02.198642    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-w2nmc" podUID="e6319ad1-4be9-44bb-898a-139e04529f67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 06:16:02.240477 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:16:02.240095    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-cd9pn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 06:16:02.240477 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:16:02.240166    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cd9pn" podUID="ce6f3d72-c7f9-43a9-83ce-40ab705d092a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 06:16:02.261334 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:16:02.261291    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-c6zlb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 06:16:02.261758 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:16:02.261727    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c6zlb" podUID="d1d4ddb1-fd17-4ac1-ada4-f785008a0913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 06:16:06.374283 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:16:06.374245    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:16:06.411516 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:16:06.411490    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:16:11.114348 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:16:11.114289    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-f6cbc132-kserve-6948cd4f9c-zlqx8" podUID="f2276668-6bb8-4a7b-8502-d83ff5667027" containerName="main" containerID="cri-o://18588daff0988ea4c966869277b3b48516badc635999f1d225fe6e1994a4091d" gracePeriod=45
Nov 01 06:16:11.232878 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:16:11.232847    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-b-f6cbc132-kserve-6948cd4f9c-zlqx8" podUID="f2276668-6bb8-4a7b-8502-d83ff5667027" containerName="main" probeResult="failure" output="Get \"https://10.134.0.154:8000/ready\": dial tcp 10.134.0.154:8000: connect: connection refused"
Nov 01 06:16:11.363532 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:16:11.363510    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-f6cbc132-kserve-6948cd4f9c-zlqx8"
Nov 01 06:16:11.484471 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:16:11.484422    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2276668-6bb8-4a7b-8502-d83ff5667027-tls-certs\" (UniqueName: \"kubernetes.io/secret/f2276668-6bb8-4a7b-8502-d83ff5667027-tls-certs\") pod \"f2276668-6bb8-4a7b-8502-d83ff5667027\" (UID: \"f2276668-6bb8-4a7b-8502-d83ff5667027\") "
Nov 01 06:16:11.484650 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:16:11.484499    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2276668-6bb8-4a7b-8502-d83ff5667027-kube-api-access-vlxr8\" (UniqueName: \"kubernetes.io/projected/f2276668-6bb8-4a7b-8502-d83ff5667027-kube-api-access-vlxr8\") pod \"f2276668-6bb8-4a7b-8502-d83ff5667027\" (UID: \"f2276668-6bb8-4a7b-8502-d83ff5667027\") "
Nov 01 06:16:11.484650 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:16:11.484532    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2276668-6bb8-4a7b-8502-d83ff5667027-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2276668-6bb8-4a7b-8502-d83ff5667027-model-cache\") pod \"f2276668-6bb8-4a7b-8502-d83ff5667027\" (UID: \"f2276668-6bb8-4a7b-8502-d83ff5667027\") "
Nov 01 06:16:11.484780 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:16:11.484695    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2276668-6bb8-4a7b-8502-d83ff5667027-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2276668-6bb8-4a7b-8502-d83ff5667027-tmp-dir\") pod \"f2276668-6bb8-4a7b-8502-d83ff5667027\" (UID: \"f2276668-6bb8-4a7b-8502-d83ff5667027\") "
Nov 01 06:16:11.484780 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:16:11.484726    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2276668-6bb8-4a7b-8502-d83ff5667027-home\" (UniqueName: \"kubernetes.io/empty-dir/f2276668-6bb8-4a7b-8502-d83ff5667027-home\") pod \"f2276668-6bb8-4a7b-8502-d83ff5667027\" (UID: \"f2276668-6bb8-4a7b-8502-d83ff5667027\") "
Nov 01 06:16:11.484780 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:16:11.484752    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2276668-6bb8-4a7b-8502-d83ff5667027-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2276668-6bb8-4a7b-8502-d83ff5667027-dshm\") pod \"f2276668-6bb8-4a7b-8502-d83ff5667027\" (UID: \"f2276668-6bb8-4a7b-8502-d83ff5667027\") "
Nov 01 06:16:11.484954 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:16:11.484864    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2276668-6bb8-4a7b-8502-d83ff5667027-model-cache" pod "f2276668-6bb8-4a7b-8502-d83ff5667027" (UID: "f2276668-6bb8-4a7b-8502-d83ff5667027"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:16:11.485010 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:16:11.484955    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2276668-6bb8-4a7b-8502-d83ff5667027-tmp-dir" pod "f2276668-6bb8-4a7b-8502-d83ff5667027" (UID: "f2276668-6bb8-4a7b-8502-d83ff5667027"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:16:11.485184 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:16:11.485159    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2276668-6bb8-4a7b-8502-d83ff5667027-home" pod "f2276668-6bb8-4a7b-8502-d83ff5667027" (UID: "f2276668-6bb8-4a7b-8502-d83ff5667027"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:16:11.485481 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:16:11.485386    5689 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2276668-6bb8-4a7b-8502-d83ff5667027-tmp-dir\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:16:11.485481 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:16:11.485411    5689 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2276668-6bb8-4a7b-8502-d83ff5667027-home\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:16:11.485481 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:16:11.485426    5689 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2276668-6bb8-4a7b-8502-d83ff5667027-model-cache\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:16:11.486885 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:16:11.486862    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2276668-6bb8-4a7b-8502-d83ff5667027-tls-certs" pod "f2276668-6bb8-4a7b-8502-d83ff5667027" (UID: "f2276668-6bb8-4a7b-8502-d83ff5667027"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:16:11.487035 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:16:11.487017    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2276668-6bb8-4a7b-8502-d83ff5667027-dshm" pod "f2276668-6bb8-4a7b-8502-d83ff5667027" (UID: "f2276668-6bb8-4a7b-8502-d83ff5667027"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:16:11.487122 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:16:11.487100    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2276668-6bb8-4a7b-8502-d83ff5667027-kube-api-access-vlxr8" pod "f2276668-6bb8-4a7b-8502-d83ff5667027" (UID: "f2276668-6bb8-4a7b-8502-d83ff5667027"). InnerVolumeSpecName "kube-api-access-vlxr8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:16:11.567632 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:16:11.567601    5689 generic.go:356] "Generic (PLEG): container finished" podID="f2276668-6bb8-4a7b-8502-d83ff5667027" containerID="18588daff0988ea4c966869277b3b48516badc635999f1d225fe6e1994a4091d" exitCode=0
Nov 01 06:16:11.567756 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:16:11.567645    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-f6cbc132-kserve-6948cd4f9c-zlqx8"
Nov 01 06:16:11.567756 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:16:11.567679    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-f6cbc132-kserve-6948cd4f9c-zlqx8" event={"ID":"f2276668-6bb8-4a7b-8502-d83ff5667027","Type":"ContainerDied","Data":"18588daff0988ea4c966869277b3b48516badc635999f1d225fe6e1994a4091d"}
Nov 01 06:16:11.567756 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:16:11.567720    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-f6cbc132-kserve-6948cd4f9c-zlqx8" event={"ID":"f2276668-6bb8-4a7b-8502-d83ff5667027","Type":"ContainerDied","Data":"6a1989aa7717a910f98d3fe020e6c9a01ff63df7d734decd8a7c171d9d21a7a1"}
Nov 01 06:16:11.567756 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:16:11.567736    5689 scope.go:122] "RemoveContainer" containerID="18588daff0988ea4c966869277b3b48516badc635999f1d225fe6e1994a4091d"
Nov 01 06:16:11.579283 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:16:11.577582    5689 scope.go:122] "RemoveContainer" containerID="18588daff0988ea4c966869277b3b48516badc635999f1d225fe6e1994a4091d"
Nov 01 06:16:11.579283 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:16:11.577886    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18588daff0988ea4c966869277b3b48516badc635999f1d225fe6e1994a4091d\": container with ID starting with 18588daff0988ea4c966869277b3b48516badc635999f1d225fe6e1994a4091d not found: ID does not exist" containerID="18588daff0988ea4c966869277b3b48516badc635999f1d225fe6e1994a4091d"
Nov 01 06:16:11.579283 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:16:11.577917    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18588daff0988ea4c966869277b3b48516badc635999f1d225fe6e1994a4091d"} err="failed to get container status \"18588daff0988ea4c966869277b3b48516badc635999f1d225fe6e1994a4091d\": rpc error: code = NotFound desc = could not find container \"18588daff0988ea4c966869277b3b48516badc635999f1d225fe6e1994a4091d\": container with ID starting with 18588daff0988ea4c966869277b3b48516badc635999f1d225fe6e1994a4091d not found: ID does not exist"
Nov 01 06:16:11.586656 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:16:11.586637    5689 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2276668-6bb8-4a7b-8502-d83ff5667027-dshm\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:16:11.586718 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:16:11.586659    5689 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2276668-6bb8-4a7b-8502-d83ff5667027-tls-certs\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:16:11.586718 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:16:11.586670    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlxr8\" (UniqueName: \"kubernetes.io/projected/f2276668-6bb8-4a7b-8502-d83ff5667027-kube-api-access-vlxr8\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:16:11.730314 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:16:11.730284    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-f6cbc132-kserve-6948cd4f9c-zlqx8"]
Nov 01 06:16:11.734733 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:16:11.734676    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-f6cbc132-kserve-6948cd4f9c-zlqx8"]
Nov 01 06:16:12.066233 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:16:12.066161    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2276668-6bb8-4a7b-8502-d83ff5667027" path="/var/lib/kubelet/pods/f2276668-6bb8-4a7b-8502-d83ff5667027/volumes"
Nov 01 06:16:15.269561 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:16:15.269532    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:16:16.738896 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:16:16.738845    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:16:17.195594 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:16:17.195552    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-w2nmc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 06:16:17.195886 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:16:17.195854    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-w2nmc" podUID="e6319ad1-4be9-44bb-898a-139e04529f67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 06:16:17.243591 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:16:17.243540    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-cd9pn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 06:16:17.243902 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:16:17.243852    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cd9pn" podUID="ce6f3d72-c7f9-43a9-83ce-40ab705d092a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 06:16:17.268901 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:16:17.268861    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-c6zlb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 06:16:17.269149 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:16:17.269112    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c6zlb" podUID="d1d4ddb1-fd17-4ac1-ada4-f785008a0913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 06:16:18.523968 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:16:18.523937    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:16:23.482437 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:16:23.482406    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:16:30.067937 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:16:30.067902    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 06:16:38.300119 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:16:38.300083    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:17:00.072788 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:17:00.072743    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 06:17:18.889901 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:17:18.889861    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:17:27.426382 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:17:27.426306    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:17:27.920395 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:17:27.920363    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:17:30.070386 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:17:30.070351    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 06:17:30.265606 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:17:30.265579    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:17:32.192373 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:17:32.192338    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:18:00.068994 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:18:00.068958    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 06:18:19.218416 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:18:19.218379    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:18:30.068140 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:18:30.068108    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 06:19:00.068800 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:19:00.068709    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 06:19:11.285646 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:19:11.285612    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-d8j7c"]
Nov 01 06:19:11.286245 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:19:11.286185    5689 cpu_manager.go:414] "containerMap: removing container" podUID="f2276668-6bb8-4a7b-8502-d83ff5667027" containerName="main"
Nov 01 06:19:11.286245 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:19:11.286205    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2276668-6bb8-4a7b-8502-d83ff5667027" containerName="main"
Nov 01 06:19:11.286368 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:19:11.286339    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2276668-6bb8-4a7b-8502-d83ff5667027" containerName="main"
Nov 01 06:19:11.289422 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:19:11.289399    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-d8j7c"
Nov 01 06:19:11.390051 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:19:11.390017    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7cbb8de-b347-4955-996b-dab57c5cbeea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-d8j7c\" (UID: \"d7cbb8de-b347-4955-996b-dab57c5cbeea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-d8j7c"
Nov 01 06:19:11.390051 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:19:11.390050    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbpbj\" (UniqueName: \"kubernetes.io/projected/d7cbb8de-b347-4955-996b-dab57c5cbeea-kube-api-access-sbpbj\") pod \"maas-api-revoke-sub-e2547622-d8j7c\" (UID: \"d7cbb8de-b347-4955-996b-dab57c5cbeea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-d8j7c"
Nov 01 06:19:11.390289 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:19:11.390263    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7cbb8de-b347-4955-996b-dab57c5cbeea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-d8j7c\" (UID: \"d7cbb8de-b347-4955-996b-dab57c5cbeea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-d8j7c"
Nov 01 06:19:11.391252 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:19:11.391230    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 06:19:11.391252 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:19:11.391235    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qtjcn\""
Nov 01 06:19:11.392664 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:19:11.392643    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-d8j7c"]
Nov 01 06:19:11.491033 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:19:11.491002    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7cbb8de-b347-4955-996b-dab57c5cbeea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-d8j7c\" (UID: \"d7cbb8de-b347-4955-996b-dab57c5cbeea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-d8j7c"
Nov 01 06:19:11.491187 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:19:11.491066    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7cbb8de-b347-4955-996b-dab57c5cbeea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-d8j7c\" (UID: \"d7cbb8de-b347-4955-996b-dab57c5cbeea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-d8j7c"
Nov 01 06:19:11.491187 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:19:11.491089    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbpbj\" (UniqueName: \"kubernetes.io/projected/d7cbb8de-b347-4955-996b-dab57c5cbeea-kube-api-access-sbpbj\") pod \"maas-api-revoke-sub-e2547622-d8j7c\" (UID: \"d7cbb8de-b347-4955-996b-dab57c5cbeea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-d8j7c"
Nov 01 06:19:11.491717 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:19:11.491695    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7cbb8de-b347-4955-996b-dab57c5cbeea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-d8j7c\" (UID: \"d7cbb8de-b347-4955-996b-dab57c5cbeea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-d8j7c"
Nov 01 06:19:11.493545 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:19:11.493524    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7cbb8de-b347-4955-996b-dab57c5cbeea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-d8j7c\" (UID: \"d7cbb8de-b347-4955-996b-dab57c5cbeea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-d8j7c"
Nov 01 06:19:11.551887 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:19:11.551836    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbpbj\" (UniqueName: \"kubernetes.io/projected/d7cbb8de-b347-4955-996b-dab57c5cbeea-kube-api-access-sbpbj\") pod \"maas-api-revoke-sub-e2547622-d8j7c\" (UID: \"d7cbb8de-b347-4955-996b-dab57c5cbeea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-d8j7c"
Nov 01 06:19:11.601400 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:19:11.601378    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-d8j7c"
Nov 01 06:19:11.972061 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:19:11.972035    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-d8j7c"]
Nov 01 06:19:11.975146 ip-10-0-129-92 kubenswrapper[5689]: W1101 06:19:11.975115    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7cbb8de_b347_4955_996b_dab57c5cbeea.slice/crio-85efc39e657fc059265545265120cec24f0a479551b6c8db41b94e7ab40e0c92 WatchSource:0}: Error finding container 85efc39e657fc059265545265120cec24f0a479551b6c8db41b94e7ab40e0c92: Status 404 returned error can't find the container with id 85efc39e657fc059265545265120cec24f0a479551b6c8db41b94e7ab40e0c92
Nov 01 06:19:12.444301 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:19:12.444256    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-d8j7c" event={"ID":"d7cbb8de-b347-4955-996b-dab57c5cbeea","Type":"ContainerStarted","Data":"20eb91249866b1afb27a12b8727fe8bff95da826719ff7694d2db3a0152acde9"}
Nov 01 06:19:12.444808 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:19:12.444355    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-d8j7c" event={"ID":"d7cbb8de-b347-4955-996b-dab57c5cbeea","Type":"ContainerStarted","Data":"85efc39e657fc059265545265120cec24f0a479551b6c8db41b94e7ab40e0c92"}
Nov 01 06:19:12.560084 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:19:12.559978    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-d8j7c" podStartSLOduration=1.559964319 podStartE2EDuration="1.559964319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:19:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:19:12.558838864 +0000 UTC m=+4530.629635581" watchObservedRunningTime="2026-11-01 06:19:12.559964319 +0000 UTC m=+4530.630761039"
Nov 01 06:19:13.450984 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:19:13.450947    5689 generic.go:356] "Generic (PLEG): container finished" podID="d7cbb8de-b347-4955-996b-dab57c5cbeea" containerID="20eb91249866b1afb27a12b8727fe8bff95da826719ff7694d2db3a0152acde9" exitCode=0
Nov 01 06:19:13.451428 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:19:13.451050    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-d8j7c" event={"ID":"d7cbb8de-b347-4955-996b-dab57c5cbeea","Type":"ContainerDied","Data":"20eb91249866b1afb27a12b8727fe8bff95da826719ff7694d2db3a0152acde9"}
Nov 01 06:19:14.589003 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:19:14.588976    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-d8j7c"
Nov 01 06:19:14.618223 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:19:14.618196    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7cbb8de-b347-4955-996b-dab57c5cbeea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7cbb8de-b347-4955-996b-dab57c5cbeea-maas-api-openshift-ca\") pod \"d7cbb8de-b347-4955-996b-dab57c5cbeea\" (UID: \"d7cbb8de-b347-4955-996b-dab57c5cbeea\") "
Nov 01 06:19:14.618357 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:19:14.618332    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7cbb8de-b347-4955-996b-dab57c5cbeea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7cbb8de-b347-4955-996b-dab57c5cbeea-maas-api-xks-ca\") pod \"d7cbb8de-b347-4955-996b-dab57c5cbeea\" (UID: \"d7cbb8de-b347-4955-996b-dab57c5cbeea\") "
Nov 01 06:19:14.618444 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:19:14.618429    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7cbb8de-b347-4955-996b-dab57c5cbeea-kube-api-access-sbpbj\" (UniqueName: \"kubernetes.io/projected/d7cbb8de-b347-4955-996b-dab57c5cbeea-kube-api-access-sbpbj\") pod \"d7cbb8de-b347-4955-996b-dab57c5cbeea\" (UID: \"d7cbb8de-b347-4955-996b-dab57c5cbeea\") "
Nov 01 06:19:14.618546 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:19:14.618507    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7cbb8de-b347-4955-996b-dab57c5cbeea-maas-api-openshift-ca" pod "d7cbb8de-b347-4955-996b-dab57c5cbeea" (UID: "d7cbb8de-b347-4955-996b-dab57c5cbeea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:19:14.618727 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:19:14.618710    5689 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7cbb8de-b347-4955-996b-dab57c5cbeea-maas-api-openshift-ca\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:19:14.620460 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:19:14.620431    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7cbb8de-b347-4955-996b-dab57c5cbeea-maas-api-xks-ca" pod "d7cbb8de-b347-4955-996b-dab57c5cbeea" (UID: "d7cbb8de-b347-4955-996b-dab57c5cbeea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:19:14.620665 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:19:14.620640    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7cbb8de-b347-4955-996b-dab57c5cbeea-kube-api-access-sbpbj" pod "d7cbb8de-b347-4955-996b-dab57c5cbeea" (UID: "d7cbb8de-b347-4955-996b-dab57c5cbeea"). InnerVolumeSpecName "kube-api-access-sbpbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:19:14.720509 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:19:14.720425    5689 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7cbb8de-b347-4955-996b-dab57c5cbeea-maas-api-xks-ca\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:19:14.720509 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:19:14.720481    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbpbj\" (UniqueName: \"kubernetes.io/projected/d7cbb8de-b347-4955-996b-dab57c5cbeea-kube-api-access-sbpbj\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:19:15.462609 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:19:15.462578    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-d8j7c"
Nov 01 06:19:15.462808 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:19:15.462578    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-d8j7c" event={"ID":"d7cbb8de-b347-4955-996b-dab57c5cbeea","Type":"ContainerDied","Data":"85efc39e657fc059265545265120cec24f0a479551b6c8db41b94e7ab40e0c92"}
Nov 01 06:19:15.462808 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:19:15.462699    5689 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85efc39e657fc059265545265120cec24f0a479551b6c8db41b94e7ab40e0c92"
Nov 01 06:19:19.649939 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:19:19.649904    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:19:23.007552 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:19:23.007515    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:19:23.181750 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:19:23.181716    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:19:23.592198 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:19:23.592158    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:19:25.905340 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:19:25.905312    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:19:30.069705 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:19:30.069669    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 06:19:36.174257 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:19:36.174223    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:19:54.505434 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:19:54.505399    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:20:00.067892 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:20:00.067853    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 06:20:11.298001 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:20:11.297968    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:20:19.929405 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:20:19.929369    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:20:30.072024 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:20:30.067380    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 06:20:42.073228 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:20:42.073198    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:20:52.679767 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:20:52.679736    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:21:00.080341 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:21:00.080302    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 06:21:05.088835 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:21:05.088804    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:21:10.649438 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:21:10.649394    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:21:10.829683 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:21:10.829656    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:21:20.246888 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:21:20.246855    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:21:20.422148 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:21:20.422115    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:21:27.592079 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:21:27.592047    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:21:30.068863 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:21:30.068829    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 06:21:35.684922 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:21:35.684888    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:21:37.080897 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:21:37.080859    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:21:45.503581 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:21:45.503547    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:21:55.492448 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:21:55.492372    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:22:00.068717 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:22:00.068682    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 06:22:19.163248 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:22:19.163211    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:22:20.599176 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:22:20.599099    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:22:29.775011 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:22:29.774979    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:22:30.068049 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:22:30.067951    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 06:22:47.786377 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:22:47.786340    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:22:58.814345 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:22:58.814315    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:23:00.068332 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:23:00.068293    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 06:23:12.066942 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:23:12.066912    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:23:20.944269 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:23:20.944194    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:23:22.004723 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:23:22.004688    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:23:30.161191 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:23:30.161149    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 06:23:33.770444 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:23:33.770408    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:23:55.607464 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:23:55.607427    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:24:00.069041 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:24:00.069002    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 06:24:06.208585 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:24:06.208549    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:24:15.045987 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:24:15.045947    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-d8j7c"]
Nov 01 06:24:15.095222 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:24:15.095187    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-d8j7c"]
Nov 01 06:24:16.066591 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:24:16.066559    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7cbb8de-b347-4955-996b-dab57c5cbeea" path="/var/lib/kubelet/pods/d7cbb8de-b347-4955-996b-dab57c5cbeea/volumes"
Nov 01 06:24:21.334441 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:24:21.334404    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:24:30.068787 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:24:30.068753    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 06:24:39.892306 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:24:39.892261    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:24:54.489129 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:24:54.489090    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:24:55.978708 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:24:55.978663    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:24:56.599486 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:24:56.587513    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:24:56.954840 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:24:56.954807    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:25:00.068837 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:25:00.068800    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 06:25:01.012871 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:25:01.012840    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:25:01.195066 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:25:01.195036    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:25:04.262229 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:25:04.262197    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:25:04.442145 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:25:04.442111    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:25:04.789904 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:25:04.789876    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-42db1c-85c47756chhgcq"]
Nov 01 06:25:04.790325 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:25:04.790312    5689 cpu_manager.go:414] "containerMap: removing container" podUID="d7cbb8de-b347-4955-996b-dab57c5cbeea" containerName="revoke-keys"
Nov 01 06:25:04.790372 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:25:04.790326    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7cbb8de-b347-4955-996b-dab57c5cbeea" containerName="revoke-keys"
Nov 01 06:25:04.790417 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:25:04.790393    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7cbb8de-b347-4955-996b-dab57c5cbeea" containerName="revoke-keys"
Nov 01 06:25:04.793402 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:25:04.793385    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-42db1c-85c47756chhgcq"
Nov 01 06:25:04.845501 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:25:04.845472    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-42db1c\""
Nov 01 06:25:04.890125 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:25:04.890088    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8e5eb5f-a9cf-458c-9354-99e8d67284e9-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-42db1c-85c47756chhgcq\" (UID: \"d8e5eb5f-a9cf-458c-9354-99e8d67284e9\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-42db1c-85c47756chhgcq"
Nov 01 06:25:04.890262 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:25:04.890154    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcgwv\" (UniqueName: \"kubernetes.io/projected/d8e5eb5f-a9cf-458c-9354-99e8d67284e9-kube-api-access-fcgwv\") pod \"payload-processing-e2e-worker-single-main-42db1c-85c47756chhgcq\" (UID: \"d8e5eb5f-a9cf-458c-9354-99e8d67284e9\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-42db1c-85c47756chhgcq"
Nov 01 06:25:04.895241 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:25:04.895221    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-4fba256d-dockercfg-d58wn\""
Nov 01 06:25:04.896599 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:25:04.896582    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-42db1c-85c47756chhgcq"]
Nov 01 06:25:04.991588 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:25:04.991557    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8e5eb5f-a9cf-458c-9354-99e8d67284e9-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-42db1c-85c47756chhgcq\" (UID: \"d8e5eb5f-a9cf-458c-9354-99e8d67284e9\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-42db1c-85c47756chhgcq"
Nov 01 06:25:04.991734 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:25:04.991618    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcgwv\" (UniqueName: \"kubernetes.io/projected/d8e5eb5f-a9cf-458c-9354-99e8d67284e9-kube-api-access-fcgwv\") pod \"payload-processing-e2e-worker-single-main-42db1c-85c47756chhgcq\" (UID: \"d8e5eb5f-a9cf-458c-9354-99e8d67284e9\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-42db1c-85c47756chhgcq"
Nov 01 06:25:04.992348 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:25:04.992328    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8e5eb5f-a9cf-458c-9354-99e8d67284e9-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-42db1c-85c47756chhgcq\" (UID: \"d8e5eb5f-a9cf-458c-9354-99e8d67284e9\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-42db1c-85c47756chhgcq"
Nov 01 06:25:05.052399 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:25:05.052339    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcgwv\" (UniqueName: \"kubernetes.io/projected/d8e5eb5f-a9cf-458c-9354-99e8d67284e9-kube-api-access-fcgwv\") pod \"payload-processing-e2e-worker-single-main-42db1c-85c47756chhgcq\" (UID: \"d8e5eb5f-a9cf-458c-9354-99e8d67284e9\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-42db1c-85c47756chhgcq"
Nov 01 06:25:05.106806 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:25:05.106783    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-42db1c-85c47756chhgcq"
Nov 01 06:25:05.480106 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:25:05.479943    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-42db1c-85c47756chhgcq"]
Nov 01 06:25:05.482952 ip-10-0-129-92 kubenswrapper[5689]: W1101 06:25:05.482925    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8e5eb5f_a9cf_458c_9354_99e8d67284e9.slice/crio-688fb357ebb3ac2f523e6f4ebdd28293bc22576b7a0f7e09d5f585b4abe3f26f WatchSource:0}: Error finding container 688fb357ebb3ac2f523e6f4ebdd28293bc22576b7a0f7e09d5f585b4abe3f26f: Status 404 returned error can't find the container with id 688fb357ebb3ac2f523e6f4ebdd28293bc22576b7a0f7e09d5f585b4abe3f26f
Nov 01 06:25:06.394499 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:25:06.394427    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-42db1c-85c47756chhgcq" event={"ID":"d8e5eb5f-a9cf-458c-9354-99e8d67284e9","Type":"ContainerStarted","Data":"0df9d9a989dc184f02978fbe20c77d17f58443b30ac29740d467ec0f9b157318"}
Nov 01 06:25:06.394679 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:25:06.394517    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-42db1c-85c47756chhgcq" event={"ID":"d8e5eb5f-a9cf-458c-9354-99e8d67284e9","Type":"ContainerStarted","Data":"688fb357ebb3ac2f523e6f4ebdd28293bc22576b7a0f7e09d5f585b4abe3f26f"}
Nov 01 06:25:06.394679 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:25:06.394634    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-42db1c-85c47756chhgcq"
Nov 01 06:25:06.511150 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:25:06.511093    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-42db1c-85c47756chhgcq" podStartSLOduration=2.51107678 podStartE2EDuration="2.51107678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:25:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:25:06.509336975 +0000 UTC m=+4884.580133696" watchObservedRunningTime="2026-11-01 06:25:06.51107678 +0000 UTC m=+4884.581873500"
Nov 01 06:25:07.138945 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:25:07.138911    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-42db1c-7f476c5786-6gs7k"]
Nov 01 06:25:07.142357 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:25:07.142341    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-42db1c-7f476c5786-6gs7k"
Nov 01 06:25:07.194751 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:25:07.194724    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-42db1c-serving-cert\""
Nov 01 06:25:07.214289 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:25:07.214264    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgsmx\" (UniqueName: \"kubernetes.io/projected/c85c684d-ae10-4b7a-85b9-faed479330ab-kube-api-access-wgsmx\") pod \"maas-api-e2e-worker-single-main-42db1c-7f476c5786-6gs7k\" (UID: \"c85c684d-ae10-4b7a-85b9-faed479330ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-42db1c-7f476c5786-6gs7k"
Nov 01 06:25:07.214428 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:25:07.214305    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c85c684d-ae10-4b7a-85b9-faed479330ab-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-42db1c-7f476c5786-6gs7k\" (UID: \"c85c684d-ae10-4b7a-85b9-faed479330ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-42db1c-7f476c5786-6gs7k"
Nov 01 06:25:07.214499 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:25:07.214472    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c85c684d-ae10-4b7a-85b9-faed479330ab-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-42db1c-7f476c5786-6gs7k\" (UID: \"c85c684d-ae10-4b7a-85b9-faed479330ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-42db1c-7f476c5786-6gs7k"
Nov 01 06:25:07.214550 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:25:07.214518    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c85c684d-ae10-4b7a-85b9-faed479330ab-metrics-certs\") pod \"maas-api-e2e-worker-single-main-42db1c-7f476c5786-6gs7k\" (UID: \"c85c684d-ae10-4b7a-85b9-faed479330ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-42db1c-7f476c5786-6gs7k"
Nov 01 06:25:07.244143 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:25:07.244117    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9729w\""
Nov 01 06:25:07.244268 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:25:07.244215    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 01 06:25:07.245678 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:25:07.245661    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-42db1c-7f476c5786-6gs7k"]
Nov 01 06:25:07.315638 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:25:07.315612    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c85c684d-ae10-4b7a-85b9-faed479330ab-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-42db1c-7f476c5786-6gs7k\" (UID: \"c85c684d-ae10-4b7a-85b9-faed479330ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-42db1c-7f476c5786-6gs7k"
Nov 01 06:25:07.315781 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:25:07.315656    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c85c684d-ae10-4b7a-85b9-faed479330ab-metrics-certs\") pod \"maas-api-e2e-worker-single-main-42db1c-7f476c5786-6gs7k\" (UID: \"c85c684d-ae10-4b7a-85b9-faed479330ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-42db1c-7f476c5786-6gs7k"
Nov 01 06:25:07.315781 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:25:07.315716    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgsmx\" (UniqueName: \"kubernetes.io/projected/c85c684d-ae10-4b7a-85b9-faed479330ab-kube-api-access-wgsmx\") pod \"maas-api-e2e-worker-single-main-42db1c-7f476c5786-6gs7k\" (UID: \"c85c684d-ae10-4b7a-85b9-faed479330ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-42db1c-7f476c5786-6gs7k"
Nov 01 06:25:07.315781 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:25:07.315747    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c85c684d-ae10-4b7a-85b9-faed479330ab-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-42db1c-7f476c5786-6gs7k\" (UID: \"c85c684d-ae10-4b7a-85b9-faed479330ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-42db1c-7f476c5786-6gs7k"
Nov 01 06:25:07.315905 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:25:07.315837    5689 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-42db1c-serving-cert: secret "maas-api-e2e-worker-single-main-42db1c-serving-cert" not found
Nov 01 06:25:07.315905 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:25:07.315903    5689 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c85c684d-ae10-4b7a-85b9-faed479330ab-maas-api-tls podName:c85c684d-ae10-4b7a-85b9-faed479330ab nodeName:}" failed. No retries permitted until 2026-11-01 06:25:07.815884104 +0000 UTC m=+4885.886680802 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c85c684d-ae10-4b7a-85b9-faed479330ab-maas-api-tls") pod "maas-api-e2e-worker-single-main-42db1c-7f476c5786-6gs7k" (UID: "c85c684d-ae10-4b7a-85b9-faed479330ab") : secret "maas-api-e2e-worker-single-main-42db1c-serving-cert" not found
Nov 01 06:25:07.316374 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:25:07.316356    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c85c684d-ae10-4b7a-85b9-faed479330ab-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-42db1c-7f476c5786-6gs7k\" (UID: \"c85c684d-ae10-4b7a-85b9-faed479330ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-42db1c-7f476c5786-6gs7k"
Nov 01 06:25:07.318945 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:25:07.318922    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c85c684d-ae10-4b7a-85b9-faed479330ab-metrics-certs\") pod \"maas-api-e2e-worker-single-main-42db1c-7f476c5786-6gs7k\" (UID: \"c85c684d-ae10-4b7a-85b9-faed479330ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-42db1c-7f476c5786-6gs7k"
Nov 01 06:25:07.359512 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:25:07.359438    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:25:07.377056 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:25:07.377033    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgsmx\" (UniqueName: \"kubernetes.io/projected/c85c684d-ae10-4b7a-85b9-faed479330ab-kube-api-access-wgsmx\") pod \"maas-api-e2e-worker-single-main-42db1c-7f476c5786-6gs7k\" (UID: \"c85c684d-ae10-4b7a-85b9-faed479330ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-42db1c-7f476c5786-6gs7k"
Nov 01 06:25:07.543009 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:25:07.542973    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:25:07.718909 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:25:07.718873    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:25:07.820800 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:25:07.820729    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c85c684d-ae10-4b7a-85b9-faed479330ab-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-42db1c-7f476c5786-6gs7k\" (UID: \"c85c684d-ae10-4b7a-85b9-faed479330ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-42db1c-7f476c5786-6gs7k"
Nov 01 06:25:07.823291 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:25:07.823264    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c85c684d-ae10-4b7a-85b9-faed479330ab-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-42db1c-7f476c5786-6gs7k\" (UID: \"c85c684d-ae10-4b7a-85b9-faed479330ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-42db1c-7f476c5786-6gs7k"
Nov 01 06:25:07.886663 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:25:07.886633    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:25:08.054853 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:25:08.054825    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-42db1c-7f476c5786-6gs7k"
Nov 01 06:25:08.430030 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:25:08.429998    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-42db1c-7f476c5786-6gs7k"]
Nov 01 06:25:08.433223 ip-10-0-129-92 kubenswrapper[5689]: W1101 06:25:08.433196    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc85c684d_ae10_4b7a_85b9_faed479330ab.slice/crio-2430a0cd1320b4624af64d829e9d85e5533c5c07c805a2895c0c3d90537cca60 WatchSource:0}: Error finding container 2430a0cd1320b4624af64d829e9d85e5533c5c07c805a2895c0c3d90537cca60: Status 404 returned error can't find the container with id 2430a0cd1320b4624af64d829e9d85e5533c5c07c805a2895c0c3d90537cca60
Nov 01 06:25:08.434585 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:25:08.434569    5689 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 06:25:09.409014 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:25:09.408576    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-42db1c-7f476c5786-6gs7k" event={"ID":"c85c684d-ae10-4b7a-85b9-faed479330ab","Type":"ContainerStarted","Data":"2430a0cd1320b4624af64d829e9d85e5533c5c07c805a2895c0c3d90537cca60"}
Nov 01 06:25:10.414597 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:25:10.414561    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-42db1c-7f476c5786-6gs7k" event={"ID":"c85c684d-ae10-4b7a-85b9-faed479330ab","Type":"ContainerStarted","Data":"039eccedf768e6e75782d717cbadc423883681a0526d94121ada97ee445e229d"}
Nov 01 06:25:10.415043 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:25:10.414660    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-42db1c-7f476c5786-6gs7k"
Nov 01 06:25:10.535610 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:25:10.535554    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-42db1c-7f476c5786-6gs7k" podStartSLOduration=2.56138928 podStartE2EDuration="3.535539239s" totalImagesPullingTime="974.149959ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:25:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:25:10.534735406 +0000 UTC m=+4888.605532128" watchObservedRunningTime="2026-11-01 06:25:10.535539239 +0000 UTC m=+4888.606335958"
Nov 01 06:25:12.319003 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:25:12.318962    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:25:16.424296 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:25:16.424265    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-42db1c-7f476c5786-6gs7k"
Nov 01 06:25:17.125652 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:25:17.125619    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:25:17.401797 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:25:17.401769    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-42db1c-85c47756chhgcq"
Nov 01 06:25:20.385726 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:25:20.385689    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:25:21.659517 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:25:21.659484    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:25:21.843532 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:25:21.843503    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:25:25.737361 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:25:25.737316    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:25:26.308651 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:25:26.308615    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:25:26.488385 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:25:26.488338    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:25:26.905890 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:25:26.905843    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:25:28.897643 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:25:28.897607    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:25:30.069551 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:25:30.069515    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 06:25:31.623644 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:25:31.623607    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:25:33.458748 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:25:33.458718    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:25:36.423983 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:25:36.423948    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:25:41.123147 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:25:41.123100    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:25:43.656718 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:25:43.656690    5689 scope.go:122] "RemoveContainer" containerID="20eb91249866b1afb27a12b8727fe8bff95da826719ff7694d2db3a0152acde9"
Nov 01 06:25:45.912988 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:25:45.912937    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:25:49.107889 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:25:49.107857    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:25:50.656688 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:25:50.656619    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:25:55.428908 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:25:55.428865    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:26:00.069207 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:26:00.069172    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 06:26:00.131347 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:26:00.131316    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:26:04.919170 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:26:04.919135    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:26:09.610708 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:26:09.610679    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:26:14.413511 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:26:14.413439    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:26:15.303278 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:26:15.303238    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:26:15.763269 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:26:15.763219    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:26:18.496315 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:26:18.496284    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:26:19.196976 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:26:19.196939    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:26:20.325845 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:26:20.325618    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:26:21.757617 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:26:21.757582    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:26:21.972678 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:26:21.972651    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:26:22.170102 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:26:22.170063    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-42db1c-85c47756chhgcq"]
Nov 01 06:26:22.575340 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:26:22.575253    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:26:22.750031 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:26:22.749982    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:26:23.369363 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:26:23.369335    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-42db1c-7f476c5786-6gs7k"]
Nov 01 06:26:23.369901 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:26:23.369610    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-42db1c-7f476c5786-6gs7k" podUID="c85c684d-ae10-4b7a-85b9-faed479330ab" containerName="maas-api" containerID="cri-o://039eccedf768e6e75782d717cbadc423883681a0526d94121ada97ee445e229d" gracePeriod=30
Nov 01 06:26:23.504598 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:26:23.504567    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:26:24.621628 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:26:24.621604    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-42db1c-7f476c5786-6gs7k"
Nov 01 06:26:24.667746 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:26:24.667720    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c85c684d-ae10-4b7a-85b9-faed479330ab-kube-api-access-wgsmx\" (UniqueName: \"kubernetes.io/projected/c85c684d-ae10-4b7a-85b9-faed479330ab-kube-api-access-wgsmx\") pod \"c85c684d-ae10-4b7a-85b9-faed479330ab\" (UID: \"c85c684d-ae10-4b7a-85b9-faed479330ab\") "
Nov 01 06:26:24.667854 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:26:24.667770    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c85c684d-ae10-4b7a-85b9-faed479330ab-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c85c684d-ae10-4b7a-85b9-faed479330ab-maas-api-tls\") pod \"c85c684d-ae10-4b7a-85b9-faed479330ab\" (UID: \"c85c684d-ae10-4b7a-85b9-faed479330ab\") "
Nov 01 06:26:24.667916 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:26:24.667905    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c85c684d-ae10-4b7a-85b9-faed479330ab-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c85c684d-ae10-4b7a-85b9-faed479330ab-metrics-certs\") pod \"c85c684d-ae10-4b7a-85b9-faed479330ab\" (UID: \"c85c684d-ae10-4b7a-85b9-faed479330ab\") "
Nov 01 06:26:24.667957 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:26:24.667930    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c85c684d-ae10-4b7a-85b9-faed479330ab-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c85c684d-ae10-4b7a-85b9-faed479330ab-openshift-service-ca\") pod \"c85c684d-ae10-4b7a-85b9-faed479330ab\" (UID: \"c85c684d-ae10-4b7a-85b9-faed479330ab\") "
Nov 01 06:26:24.668354 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:26:24.668327    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c85c684d-ae10-4b7a-85b9-faed479330ab-openshift-service-ca" pod "c85c684d-ae10-4b7a-85b9-faed479330ab" (UID: "c85c684d-ae10-4b7a-85b9-faed479330ab"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:26:24.669914 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:26:24.669893    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c85c684d-ae10-4b7a-85b9-faed479330ab-kube-api-access-wgsmx" pod "c85c684d-ae10-4b7a-85b9-faed479330ab" (UID: "c85c684d-ae10-4b7a-85b9-faed479330ab"). InnerVolumeSpecName "kube-api-access-wgsmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:26:24.670009 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:26:24.669922    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c85c684d-ae10-4b7a-85b9-faed479330ab-maas-api-tls" pod "c85c684d-ae10-4b7a-85b9-faed479330ab" (UID: "c85c684d-ae10-4b7a-85b9-faed479330ab"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:26:24.670095 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:26:24.670047    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c85c684d-ae10-4b7a-85b9-faed479330ab-metrics-certs" pod "c85c684d-ae10-4b7a-85b9-faed479330ab" (UID: "c85c684d-ae10-4b7a-85b9-faed479330ab"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:26:24.760164 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:26:24.760084    5689 generic.go:356] "Generic (PLEG): container finished" podID="c85c684d-ae10-4b7a-85b9-faed479330ab" containerID="039eccedf768e6e75782d717cbadc423883681a0526d94121ada97ee445e229d" exitCode=0
Nov 01 06:26:24.760164 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:26:24.760131    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-42db1c-7f476c5786-6gs7k"
Nov 01 06:26:24.760164 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:26:24.760154    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-42db1c-7f476c5786-6gs7k" event={"ID":"c85c684d-ae10-4b7a-85b9-faed479330ab","Type":"ContainerDied","Data":"039eccedf768e6e75782d717cbadc423883681a0526d94121ada97ee445e229d"}
Nov 01 06:26:24.760396 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:26:24.760204    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-42db1c-7f476c5786-6gs7k" event={"ID":"c85c684d-ae10-4b7a-85b9-faed479330ab","Type":"ContainerDied","Data":"2430a0cd1320b4624af64d829e9d85e5533c5c07c805a2895c0c3d90537cca60"}
Nov 01 06:26:24.760396 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:26:24.760232    5689 scope.go:122] "RemoveContainer" containerID="039eccedf768e6e75782d717cbadc423883681a0526d94121ada97ee445e229d"
Nov 01 06:26:24.768822 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:26:24.768790    5689 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c85c684d-ae10-4b7a-85b9-faed479330ab-metrics-certs\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:24.768938 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:26:24.768826    5689 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c85c684d-ae10-4b7a-85b9-faed479330ab-openshift-service-ca\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:24.768938 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:26:24.768840    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgsmx\" (UniqueName: \"kubernetes.io/projected/c85c684d-ae10-4b7a-85b9-faed479330ab-kube-api-access-wgsmx\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:24.768938 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:26:24.768855    5689 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c85c684d-ae10-4b7a-85b9-faed479330ab-maas-api-tls\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:24.771034 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:26:24.771008    5689 scope.go:122] "RemoveContainer" containerID="039eccedf768e6e75782d717cbadc423883681a0526d94121ada97ee445e229d"
Nov 01 06:26:24.771300 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:26:24.771278    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"039eccedf768e6e75782d717cbadc423883681a0526d94121ada97ee445e229d\": container with ID starting with 039eccedf768e6e75782d717cbadc423883681a0526d94121ada97ee445e229d not found: ID does not exist" containerID="039eccedf768e6e75782d717cbadc423883681a0526d94121ada97ee445e229d"
Nov 01 06:26:24.771362 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:26:24.771309    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"039eccedf768e6e75782d717cbadc423883681a0526d94121ada97ee445e229d"} err="failed to get container status \"039eccedf768e6e75782d717cbadc423883681a0526d94121ada97ee445e229d\": rpc error: code = NotFound desc = could not find container \"039eccedf768e6e75782d717cbadc423883681a0526d94121ada97ee445e229d\": container with ID starting with 039eccedf768e6e75782d717cbadc423883681a0526d94121ada97ee445e229d not found: ID does not exist"
Nov 01 06:26:24.921740 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:26:24.921709    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-42db1c-7f476c5786-6gs7k"]
Nov 01 06:26:24.927326 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:26:24.927302    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-42db1c-7f476c5786-6gs7k"]
Nov 01 06:26:26.066020 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:26:26.065980    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c85c684d-ae10-4b7a-85b9-faed479330ab" path="/var/lib/kubelet/pods/c85c684d-ae10-4b7a-85b9-faed479330ab/volumes"
Nov 01 06:26:27.171343 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:26:27.171284    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-42db1c-85c47756chhgcq" podUID="d8e5eb5f-a9cf-458c-9354-99e8d67284e9" containerName="payload-processing" containerID="cri-o://0df9d9a989dc184f02978fbe20c77d17f58443b30ac29740d467ec0f9b157318" gracePeriod=25
Nov 01 06:26:27.291417 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:26:27.291383    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:26:27.400725 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:26:27.400683    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-42db1c-85c47756chhgcq" podUID="d8e5eb5f-a9cf-458c-9354-99e8d67284e9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:26:27.475607 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:26:27.475530    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:26:30.069630 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:26:30.069595    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 06:26:32.359132 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:26:32.359094    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:26:37.400472 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:26:37.400410    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-42db1c-85c47756chhgcq" podUID="d8e5eb5f-a9cf-458c-9354-99e8d67284e9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:26:47.424573 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:26:47.424549    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-42db1c-85c47756chhgcq"
Nov 01 06:26:47.572505 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:26:47.572424    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8e5eb5f-a9cf-458c-9354-99e8d67284e9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8e5eb5f-a9cf-458c-9354-99e8d67284e9-plugins-config-volume\") pod \"d8e5eb5f-a9cf-458c-9354-99e8d67284e9\" (UID: \"d8e5eb5f-a9cf-458c-9354-99e8d67284e9\") "
Nov 01 06:26:47.572505 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:26:47.572500    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8e5eb5f-a9cf-458c-9354-99e8d67284e9-kube-api-access-fcgwv\" (UniqueName: \"kubernetes.io/projected/d8e5eb5f-a9cf-458c-9354-99e8d67284e9-kube-api-access-fcgwv\") pod \"d8e5eb5f-a9cf-458c-9354-99e8d67284e9\" (UID: \"d8e5eb5f-a9cf-458c-9354-99e8d67284e9\") "
Nov 01 06:26:47.572774 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:26:47.572754    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8e5eb5f-a9cf-458c-9354-99e8d67284e9-plugins-config-volume" pod "d8e5eb5f-a9cf-458c-9354-99e8d67284e9" (UID: "d8e5eb5f-a9cf-458c-9354-99e8d67284e9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:26:47.574763 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:26:47.574735    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8e5eb5f-a9cf-458c-9354-99e8d67284e9-kube-api-access-fcgwv" pod "d8e5eb5f-a9cf-458c-9354-99e8d67284e9" (UID: "d8e5eb5f-a9cf-458c-9354-99e8d67284e9"). InnerVolumeSpecName "kube-api-access-fcgwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:26:47.673507 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:26:47.673481    5689 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8e5eb5f-a9cf-458c-9354-99e8d67284e9-plugins-config-volume\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:47.673507 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:26:47.673506    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcgwv\" (UniqueName: \"kubernetes.io/projected/d8e5eb5f-a9cf-458c-9354-99e8d67284e9-kube-api-access-fcgwv\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:47.861775 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:26:47.861690    5689 generic.go:356] "Generic (PLEG): container finished" podID="d8e5eb5f-a9cf-458c-9354-99e8d67284e9" containerID="0df9d9a989dc184f02978fbe20c77d17f58443b30ac29740d467ec0f9b157318" exitCode=0
Nov 01 06:26:47.861775 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:26:47.861735    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-42db1c-85c47756chhgcq"
Nov 01 06:26:47.861970 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:26:47.861767    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-42db1c-85c47756chhgcq" event={"ID":"d8e5eb5f-a9cf-458c-9354-99e8d67284e9","Type":"ContainerDied","Data":"0df9d9a989dc184f02978fbe20c77d17f58443b30ac29740d467ec0f9b157318"}
Nov 01 06:26:47.861970 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:26:47.861824    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-42db1c-85c47756chhgcq" event={"ID":"d8e5eb5f-a9cf-458c-9354-99e8d67284e9","Type":"ContainerDied","Data":"688fb357ebb3ac2f523e6f4ebdd28293bc22576b7a0f7e09d5f585b4abe3f26f"}
Nov 01 06:26:47.861970 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:26:47.861853    5689 scope.go:122] "RemoveContainer" containerID="0df9d9a989dc184f02978fbe20c77d17f58443b30ac29740d467ec0f9b157318"
Nov 01 06:26:47.871243 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:26:47.871227    5689 scope.go:122] "RemoveContainer" containerID="0df9d9a989dc184f02978fbe20c77d17f58443b30ac29740d467ec0f9b157318"
Nov 01 06:26:47.871542 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:26:47.871516    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0df9d9a989dc184f02978fbe20c77d17f58443b30ac29740d467ec0f9b157318\": container with ID starting with 0df9d9a989dc184f02978fbe20c77d17f58443b30ac29740d467ec0f9b157318 not found: ID does not exist" containerID="0df9d9a989dc184f02978fbe20c77d17f58443b30ac29740d467ec0f9b157318"
Nov 01 06:26:47.871645 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:26:47.871553    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0df9d9a989dc184f02978fbe20c77d17f58443b30ac29740d467ec0f9b157318"} err="failed to get container status \"0df9d9a989dc184f02978fbe20c77d17f58443b30ac29740d467ec0f9b157318\": rpc error: code = NotFound desc = could not find container \"0df9d9a989dc184f02978fbe20c77d17f58443b30ac29740d467ec0f9b157318\": container with ID starting with 0df9d9a989dc184f02978fbe20c77d17f58443b30ac29740d467ec0f9b157318 not found: ID does not exist"
Nov 01 06:26:48.023663 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:26:48.023628    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-42db1c-85c47756chhgcq"]
Nov 01 06:26:48.073022 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:26:48.072998    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-42db1c-85c47756chhgcq"]
Nov 01 06:26:48.400075 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:26:48.400037    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-42db1c-85c47756chhgcq" podUID="d8e5eb5f-a9cf-458c-9354-99e8d67284e9" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.156:50052\" within 1s: context deadline exceeded"
Nov 01 06:26:48.400223 ip-10-0-129-92 kubenswrapper[5689]: W1101 06:26:48.400126    5689 logging.go:55] [core] [Channel #5548 SubChannel #5549]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.156:50052", ServerName: "10.134.0.156:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.156:50052: operation was canceled"
Nov 01 06:26:50.065884 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:26:50.065853    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8e5eb5f-a9cf-458c-9354-99e8d67284e9" path="/var/lib/kubelet/pods/d8e5eb5f-a9cf-458c-9354-99e8d67284e9/volumes"
Nov 01 06:26:58.496729 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:26:58.496697    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:26:58.669118 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:26:58.669089    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:26:58.882855 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:26:58.882750    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:26:59.295990 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:26:59.295959    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:26:59.899423 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:26:59.899367    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:27:00.068900 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:27:00.068865    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 06:27:03.682018 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:27:03.681987    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:27:03.861822 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:27:03.861796    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:27:16.096180 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:27:16.096140    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:27:18.282071 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:27:18.281638    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-tgfm2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 06:27:18.282071 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:27:18.281752    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tgfm2" podUID="5665e928-b4df-455f-99c1-d466700fee62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 06:27:22.329938 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:27:22.329876    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:27:29.769557 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:27:29.769499    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:27:29.947663 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:27:29.947630    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:27:30.067640 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:27:30.067557    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 06:28:00.068046 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:28:00.067967    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 06:28:22.671617 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:28:22.671588    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:28:30.068176 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:28:30.068144    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 06:28:47.194597 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:28:47.194554    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-w2nmc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 06:28:47.195210 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:28:47.195174    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-w2nmc" podUID="e6319ad1-4be9-44bb-898a-139e04529f67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 06:28:47.246479 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:28:47.245549    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-cd9pn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 06:28:47.246479 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:28:47.245612    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-cd9pn" podUID="ce6f3d72-c7f9-43a9-83ce-40ab705d092a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 06:28:47.267605 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:28:47.267539    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-c6zlb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 06:28:47.267977 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:28:47.267905    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-c6zlb" podUID="d1d4ddb1-fd17-4ac1-ada4-f785008a0913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 06:28:49.517010 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:28:49.516972    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:29:00.067746 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:29:00.067712    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 06:29:04.088208 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:04.088170    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:29:12.443089 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:12.443054    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:29:20.892344 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:20.892263    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:29:23.069168 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:23.069130    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:29:25.478947 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:25.478908    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7b479cdcf5-7hkgg"]
Nov 01 06:29:25.479345 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:25.479315    5689 cpu_manager.go:414] "containerMap: removing container" podUID="d8e5eb5f-a9cf-458c-9354-99e8d67284e9" containerName="payload-processing"
Nov 01 06:29:25.479345 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:25.479327    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8e5eb5f-a9cf-458c-9354-99e8d67284e9" containerName="payload-processing"
Nov 01 06:29:25.479431 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:25.479351    5689 cpu_manager.go:414] "containerMap: removing container" podUID="c85c684d-ae10-4b7a-85b9-faed479330ab" containerName="maas-api"
Nov 01 06:29:25.479431 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:25.479356    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c85c684d-ae10-4b7a-85b9-faed479330ab" containerName="maas-api"
Nov 01 06:29:25.479431 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:25.479419    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8e5eb5f-a9cf-458c-9354-99e8d67284e9" containerName="payload-processing"
Nov 01 06:29:25.479431 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:25.479430    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="c85c684d-ae10-4b7a-85b9-faed479330ab" containerName="maas-api"
Nov 01 06:29:25.482436 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:25.482418    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7b479cdcf5-7hkgg"
Nov 01 06:29:25.495560 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:25.495530    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5cdb759fdc-ww7m7"]
Nov 01 06:29:25.499000 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:25.498981    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5cdb759fdc-ww7m7"
Nov 01 06:29:25.542608 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:25.542575    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 01 06:29:25.542608 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:25.542593    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9729w\""
Nov 01 06:29:25.565777 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:25.565745    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/262f6665-9ed2-41fc-9820-b77abf3d654d-openshift-service-ca\") pod \"maas-api-5cdb759fdc-ww7m7\" (UID: \"262f6665-9ed2-41fc-9820-b77abf3d654d\") " pod="odh-ai-gateway-infra/maas-api-5cdb759fdc-ww7m7"
Nov 01 06:29:25.565777 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:25.565777    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/262f6665-9ed2-41fc-9820-b77abf3d654d-maas-api-tls\") pod \"maas-api-5cdb759fdc-ww7m7\" (UID: \"262f6665-9ed2-41fc-9820-b77abf3d654d\") " pod="odh-ai-gateway-infra/maas-api-5cdb759fdc-ww7m7"
Nov 01 06:29:25.565938 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:25.565798    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-855l4\" (UniqueName: \"kubernetes.io/projected/70b65140-62f2-4383-a263-5ca1fc7e9a10-kube-api-access-855l4\") pod \"maas-api-7b479cdcf5-7hkgg\" (UID: \"70b65140-62f2-4383-a263-5ca1fc7e9a10\") " pod="odh-ai-gateway-infra/maas-api-7b479cdcf5-7hkgg"
Nov 01 06:29:25.565938 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:25.565919    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70b65140-62f2-4383-a263-5ca1fc7e9a10-metrics-certs\") pod \"maas-api-7b479cdcf5-7hkgg\" (UID: \"70b65140-62f2-4383-a263-5ca1fc7e9a10\") " pod="odh-ai-gateway-infra/maas-api-7b479cdcf5-7hkgg"
Nov 01 06:29:25.566009 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:25.565968    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70b65140-62f2-4383-a263-5ca1fc7e9a10-openshift-service-ca\") pod \"maas-api-7b479cdcf5-7hkgg\" (UID: \"70b65140-62f2-4383-a263-5ca1fc7e9a10\") " pod="odh-ai-gateway-infra/maas-api-7b479cdcf5-7hkgg"
Nov 01 06:29:25.566071 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:25.566051    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/262f6665-9ed2-41fc-9820-b77abf3d654d-metrics-certs\") pod \"maas-api-5cdb759fdc-ww7m7\" (UID: \"262f6665-9ed2-41fc-9820-b77abf3d654d\") " pod="odh-ai-gateway-infra/maas-api-5cdb759fdc-ww7m7"
Nov 01 06:29:25.566108 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:25.566087    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv4x2\" (UniqueName: \"kubernetes.io/projected/262f6665-9ed2-41fc-9820-b77abf3d654d-kube-api-access-vv4x2\") pod \"maas-api-5cdb759fdc-ww7m7\" (UID: \"262f6665-9ed2-41fc-9820-b77abf3d654d\") " pod="odh-ai-gateway-infra/maas-api-5cdb759fdc-ww7m7"
Nov 01 06:29:25.566150 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:25.566139    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70b65140-62f2-4383-a263-5ca1fc7e9a10-maas-api-tls\") pod \"maas-api-7b479cdcf5-7hkgg\" (UID: \"70b65140-62f2-4383-a263-5ca1fc7e9a10\") " pod="odh-ai-gateway-infra/maas-api-7b479cdcf5-7hkgg"
Nov 01 06:29:25.583657 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:25.583631    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Nov 01 06:29:25.586072 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:25.586047    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5cdb759fdc-ww7m7"]
Nov 01 06:29:25.588561 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:25.588538    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b479cdcf5-7hkgg"]
Nov 01 06:29:25.666754 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:25.666717    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70b65140-62f2-4383-a263-5ca1fc7e9a10-openshift-service-ca\") pod \"maas-api-7b479cdcf5-7hkgg\" (UID: \"70b65140-62f2-4383-a263-5ca1fc7e9a10\") " pod="odh-ai-gateway-infra/maas-api-7b479cdcf5-7hkgg"
Nov 01 06:29:25.666903 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:25.666780    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/262f6665-9ed2-41fc-9820-b77abf3d654d-metrics-certs\") pod \"maas-api-5cdb759fdc-ww7m7\" (UID: \"262f6665-9ed2-41fc-9820-b77abf3d654d\") " pod="odh-ai-gateway-infra/maas-api-5cdb759fdc-ww7m7"
Nov 01 06:29:25.666903 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:25.666816    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv4x2\" (UniqueName: \"kubernetes.io/projected/262f6665-9ed2-41fc-9820-b77abf3d654d-kube-api-access-vv4x2\") pod \"maas-api-5cdb759fdc-ww7m7\" (UID: \"262f6665-9ed2-41fc-9820-b77abf3d654d\") " pod="odh-ai-gateway-infra/maas-api-5cdb759fdc-ww7m7"
Nov 01 06:29:25.666903 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:25.666849    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70b65140-62f2-4383-a263-5ca1fc7e9a10-maas-api-tls\") pod \"maas-api-7b479cdcf5-7hkgg\" (UID: \"70b65140-62f2-4383-a263-5ca1fc7e9a10\") " pod="odh-ai-gateway-infra/maas-api-7b479cdcf5-7hkgg"
Nov 01 06:29:25.667031 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:25.666933    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/262f6665-9ed2-41fc-9820-b77abf3d654d-openshift-service-ca\") pod \"maas-api-5cdb759fdc-ww7m7\" (UID: \"262f6665-9ed2-41fc-9820-b77abf3d654d\") " pod="odh-ai-gateway-infra/maas-api-5cdb759fdc-ww7m7"
Nov 01 06:29:25.667031 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:25.666963    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/262f6665-9ed2-41fc-9820-b77abf3d654d-maas-api-tls\") pod \"maas-api-5cdb759fdc-ww7m7\" (UID: \"262f6665-9ed2-41fc-9820-b77abf3d654d\") " pod="odh-ai-gateway-infra/maas-api-5cdb759fdc-ww7m7"
Nov 01 06:29:25.667031 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:25.667001    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-855l4\" (UniqueName: \"kubernetes.io/projected/70b65140-62f2-4383-a263-5ca1fc7e9a10-kube-api-access-855l4\") pod \"maas-api-7b479cdcf5-7hkgg\" (UID: \"70b65140-62f2-4383-a263-5ca1fc7e9a10\") " pod="odh-ai-gateway-infra/maas-api-7b479cdcf5-7hkgg"
Nov 01 06:29:25.667262 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:25.667066    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70b65140-62f2-4383-a263-5ca1fc7e9a10-metrics-certs\") pod \"maas-api-7b479cdcf5-7hkgg\" (UID: \"70b65140-62f2-4383-a263-5ca1fc7e9a10\") " pod="odh-ai-gateway-infra/maas-api-7b479cdcf5-7hkgg"
Nov 01 06:29:25.667510 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:25.667487    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70b65140-62f2-4383-a263-5ca1fc7e9a10-openshift-service-ca\") pod \"maas-api-7b479cdcf5-7hkgg\" (UID: \"70b65140-62f2-4383-a263-5ca1fc7e9a10\") " pod="odh-ai-gateway-infra/maas-api-7b479cdcf5-7hkgg"
Nov 01 06:29:25.667763 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:25.667738    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/262f6665-9ed2-41fc-9820-b77abf3d654d-openshift-service-ca\") pod \"maas-api-5cdb759fdc-ww7m7\" (UID: \"262f6665-9ed2-41fc-9820-b77abf3d654d\") " pod="odh-ai-gateway-infra/maas-api-5cdb759fdc-ww7m7"
Nov 01 06:29:25.669688 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:25.669668    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/262f6665-9ed2-41fc-9820-b77abf3d654d-maas-api-tls\") pod \"maas-api-5cdb759fdc-ww7m7\" (UID: \"262f6665-9ed2-41fc-9820-b77abf3d654d\") " pod="odh-ai-gateway-infra/maas-api-5cdb759fdc-ww7m7"
Nov 01 06:29:25.670004 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:25.669982    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70b65140-62f2-4383-a263-5ca1fc7e9a10-maas-api-tls\") pod \"maas-api-7b479cdcf5-7hkgg\" (UID: \"70b65140-62f2-4383-a263-5ca1fc7e9a10\") " pod="odh-ai-gateway-infra/maas-api-7b479cdcf5-7hkgg"
Nov 01 06:29:25.670059 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:25.670035    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/262f6665-9ed2-41fc-9820-b77abf3d654d-metrics-certs\") pod \"maas-api-5cdb759fdc-ww7m7\" (UID: \"262f6665-9ed2-41fc-9820-b77abf3d654d\") " pod="odh-ai-gateway-infra/maas-api-5cdb759fdc-ww7m7"
Nov 01 06:29:25.670166 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:25.670147    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70b65140-62f2-4383-a263-5ca1fc7e9a10-metrics-certs\") pod \"maas-api-7b479cdcf5-7hkgg\" (UID: \"70b65140-62f2-4383-a263-5ca1fc7e9a10\") " pod="odh-ai-gateway-infra/maas-api-7b479cdcf5-7hkgg"
Nov 01 06:29:25.728595 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:25.728570    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv4x2\" (UniqueName: \"kubernetes.io/projected/262f6665-9ed2-41fc-9820-b77abf3d654d-kube-api-access-vv4x2\") pod \"maas-api-5cdb759fdc-ww7m7\" (UID: \"262f6665-9ed2-41fc-9820-b77abf3d654d\") " pod="odh-ai-gateway-infra/maas-api-5cdb759fdc-ww7m7"
Nov 01 06:29:25.728745 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:25.728594    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-855l4\" (UniqueName: \"kubernetes.io/projected/70b65140-62f2-4383-a263-5ca1fc7e9a10-kube-api-access-855l4\") pod \"maas-api-7b479cdcf5-7hkgg\" (UID: \"70b65140-62f2-4383-a263-5ca1fc7e9a10\") " pod="odh-ai-gateway-infra/maas-api-7b479cdcf5-7hkgg"
Nov 01 06:29:25.794964 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:25.794906    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7b479cdcf5-7hkgg"
Nov 01 06:29:25.812331 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:25.812308    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5cdb759fdc-ww7m7"
Nov 01 06:29:26.195615 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:26.195586    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b479cdcf5-7hkgg"]
Nov 01 06:29:26.198976 ip-10-0-129-92 kubenswrapper[5689]: W1101 06:29:26.198946    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70b65140_62f2_4383_a263_5ca1fc7e9a10.slice/crio-8260a5549d93ab1b95c51052b81326e179822a4faf76a7e5ddfdbea868c7909a WatchSource:0}: Error finding container 8260a5549d93ab1b95c51052b81326e179822a4faf76a7e5ddfdbea868c7909a: Status 404 returned error can't find the container with id 8260a5549d93ab1b95c51052b81326e179822a4faf76a7e5ddfdbea868c7909a
Nov 01 06:29:26.226446 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:26.226426    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5cdb759fdc-ww7m7"]
Nov 01 06:29:26.229339 ip-10-0-129-92 kubenswrapper[5689]: W1101 06:29:26.229314    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod262f6665_9ed2_41fc_9820_b77abf3d654d.slice/crio-ca1fc8fcc1ab6bc09f4d07c340a1a6e0f3d28ce422c593da2d6dba86caac5fe4 WatchSource:0}: Error finding container ca1fc8fcc1ab6bc09f4d07c340a1a6e0f3d28ce422c593da2d6dba86caac5fe4: Status 404 returned error can't find the container with id ca1fc8fcc1ab6bc09f4d07c340a1a6e0f3d28ce422c593da2d6dba86caac5fe4
Nov 01 06:29:26.625558 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:26.625434    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5cdb759fdc-ww7m7" event={"ID":"262f6665-9ed2-41fc-9820-b77abf3d654d","Type":"ContainerStarted","Data":"ca1fc8fcc1ab6bc09f4d07c340a1a6e0f3d28ce422c593da2d6dba86caac5fe4"}
Nov 01 06:29:26.626544 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:26.626517    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7b479cdcf5-7hkgg" event={"ID":"70b65140-62f2-4383-a263-5ca1fc7e9a10","Type":"ContainerStarted","Data":"8260a5549d93ab1b95c51052b81326e179822a4faf76a7e5ddfdbea868c7909a"}
Nov 01 06:29:27.632888 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:27.632836    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7b479cdcf5-7hkgg" event={"ID":"70b65140-62f2-4383-a263-5ca1fc7e9a10","Type":"ContainerStarted","Data":"9bc90a6cfd616f93eac12b088546746026ea3ccfeaead6ccfbc07c125d5b6a3c"}
Nov 01 06:29:27.633335 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:27.632950    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7b479cdcf5-7hkgg"
Nov 01 06:29:27.634479 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:27.634426    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5cdb759fdc-ww7m7" event={"ID":"262f6665-9ed2-41fc-9820-b77abf3d654d","Type":"ContainerStarted","Data":"7aee294c54427fdebdb1621ed289e38d2641eed150b93b3b2a3272cdb37d1bb3"}
Nov 01 06:29:27.634627 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:27.634562    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5cdb759fdc-ww7m7"
Nov 01 06:29:27.751048 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:27.751002    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7b479cdcf5-7hkgg" podStartSLOduration=1.948045586 podStartE2EDuration="2.7509902s" totalImagesPullingTime="802.944614ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:29:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:29:27.74930568 +0000 UTC m=+5145.820102401" watchObservedRunningTime="2026-11-01 06:29:27.7509902 +0000 UTC m=+5145.821786936"
Nov 01 06:29:27.871380 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:27.871281    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5cdb759fdc-ww7m7" podStartSLOduration=2.077503889 podStartE2EDuration="2.871264257s" totalImagesPullingTime="793.760368ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:29:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:29:27.870285446 +0000 UTC m=+5145.941082168" watchObservedRunningTime="2026-11-01 06:29:27.871264257 +0000 UTC m=+5145.942060981"
Nov 01 06:29:30.069755 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:29:30.069721    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 06:29:33.644911 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:33.644878    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7b479cdcf5-7hkgg"
Nov 01 06:29:33.645988 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:33.645965    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5cdb759fdc-ww7m7"
Nov 01 06:29:33.909419 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:33.909389    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b479cdcf5-7hkgg"]
Nov 01 06:29:33.909700 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:33.909671    5689 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-7b479cdcf5-7hkgg" podUID="70b65140-62f2-4383-a263-5ca1fc7e9a10" containerName="maas-api" containerID="cri-o://9bc90a6cfd616f93eac12b088546746026ea3ccfeaead6ccfbc07c125d5b6a3c" gracePeriod=30
Nov 01 06:29:34.153071 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:34.153045    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7b479cdcf5-7hkgg"
Nov 01 06:29:34.250524 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:34.250439    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70b65140-62f2-4383-a263-5ca1fc7e9a10-metrics-certs\" (UniqueName: \"kubernetes.io/secret/70b65140-62f2-4383-a263-5ca1fc7e9a10-metrics-certs\") pod \"70b65140-62f2-4383-a263-5ca1fc7e9a10\" (UID: \"70b65140-62f2-4383-a263-5ca1fc7e9a10\") "
Nov 01 06:29:34.250668 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:34.250522    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70b65140-62f2-4383-a263-5ca1fc7e9a10-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70b65140-62f2-4383-a263-5ca1fc7e9a10-maas-api-tls\") pod \"70b65140-62f2-4383-a263-5ca1fc7e9a10\" (UID: \"70b65140-62f2-4383-a263-5ca1fc7e9a10\") "
Nov 01 06:29:34.250747 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:34.250688    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70b65140-62f2-4383-a263-5ca1fc7e9a10-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70b65140-62f2-4383-a263-5ca1fc7e9a10-openshift-service-ca\") pod \"70b65140-62f2-4383-a263-5ca1fc7e9a10\" (UID: \"70b65140-62f2-4383-a263-5ca1fc7e9a10\") "
Nov 01 06:29:34.250747 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:34.250729    5689 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70b65140-62f2-4383-a263-5ca1fc7e9a10-kube-api-access-855l4\" (UniqueName: \"kubernetes.io/projected/70b65140-62f2-4383-a263-5ca1fc7e9a10-kube-api-access-855l4\") pod \"70b65140-62f2-4383-a263-5ca1fc7e9a10\" (UID: \"70b65140-62f2-4383-a263-5ca1fc7e9a10\") "
Nov 01 06:29:34.251075 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:34.251049    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70b65140-62f2-4383-a263-5ca1fc7e9a10-openshift-service-ca" pod "70b65140-62f2-4383-a263-5ca1fc7e9a10" (UID: "70b65140-62f2-4383-a263-5ca1fc7e9a10"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:29:34.252689 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:34.252665    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70b65140-62f2-4383-a263-5ca1fc7e9a10-maas-api-tls" pod "70b65140-62f2-4383-a263-5ca1fc7e9a10" (UID: "70b65140-62f2-4383-a263-5ca1fc7e9a10"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:29:34.252787 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:34.252709    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70b65140-62f2-4383-a263-5ca1fc7e9a10-metrics-certs" pod "70b65140-62f2-4383-a263-5ca1fc7e9a10" (UID: "70b65140-62f2-4383-a263-5ca1fc7e9a10"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:29:34.252834 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:34.252787    5689 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70b65140-62f2-4383-a263-5ca1fc7e9a10-kube-api-access-855l4" pod "70b65140-62f2-4383-a263-5ca1fc7e9a10" (UID: "70b65140-62f2-4383-a263-5ca1fc7e9a10"). InnerVolumeSpecName "kube-api-access-855l4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:29:34.351828 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:34.351792    5689 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70b65140-62f2-4383-a263-5ca1fc7e9a10-maas-api-tls\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:29:34.351828 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:34.351822    5689 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70b65140-62f2-4383-a263-5ca1fc7e9a10-openshift-service-ca\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:29:34.352009 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:34.351836    5689 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-855l4\" (UniqueName: \"kubernetes.io/projected/70b65140-62f2-4383-a263-5ca1fc7e9a10-kube-api-access-855l4\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:29:34.352009 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:34.351850    5689 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70b65140-62f2-4383-a263-5ca1fc7e9a10-metrics-certs\") on node \"ip-10-0-129-92.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:29:34.667864 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:34.667834    5689 generic.go:356] "Generic (PLEG): container finished" podID="70b65140-62f2-4383-a263-5ca1fc7e9a10" containerID="9bc90a6cfd616f93eac12b088546746026ea3ccfeaead6ccfbc07c125d5b6a3c" exitCode=0
Nov 01 06:29:34.668358 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:34.667871    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7b479cdcf5-7hkgg" event={"ID":"70b65140-62f2-4383-a263-5ca1fc7e9a10","Type":"ContainerDied","Data":"9bc90a6cfd616f93eac12b088546746026ea3ccfeaead6ccfbc07c125d5b6a3c"}
Nov 01 06:29:34.668358 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:34.667884    5689 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7b479cdcf5-7hkgg"
Nov 01 06:29:34.668358 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:34.667912    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7b479cdcf5-7hkgg" event={"ID":"70b65140-62f2-4383-a263-5ca1fc7e9a10","Type":"ContainerDied","Data":"8260a5549d93ab1b95c51052b81326e179822a4faf76a7e5ddfdbea868c7909a"}
Nov 01 06:29:34.668358 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:34.667933    5689 scope.go:122] "RemoveContainer" containerID="9bc90a6cfd616f93eac12b088546746026ea3ccfeaead6ccfbc07c125d5b6a3c"
Nov 01 06:29:34.679359 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:34.679336    5689 scope.go:122] "RemoveContainer" containerID="9bc90a6cfd616f93eac12b088546746026ea3ccfeaead6ccfbc07c125d5b6a3c"
Nov 01 06:29:34.680170 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:29:34.680139    5689 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bc90a6cfd616f93eac12b088546746026ea3ccfeaead6ccfbc07c125d5b6a3c\": container with ID starting with 9bc90a6cfd616f93eac12b088546746026ea3ccfeaead6ccfbc07c125d5b6a3c not found: ID does not exist" containerID="9bc90a6cfd616f93eac12b088546746026ea3ccfeaead6ccfbc07c125d5b6a3c"
Nov 01 06:29:34.680295 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:34.680180    5689 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bc90a6cfd616f93eac12b088546746026ea3ccfeaead6ccfbc07c125d5b6a3c"} err="failed to get container status \"9bc90a6cfd616f93eac12b088546746026ea3ccfeaead6ccfbc07c125d5b6a3c\": rpc error: code = NotFound desc = could not find container \"9bc90a6cfd616f93eac12b088546746026ea3ccfeaead6ccfbc07c125d5b6a3c\": container with ID starting with 9bc90a6cfd616f93eac12b088546746026ea3ccfeaead6ccfbc07c125d5b6a3c not found: ID does not exist"
Nov 01 06:29:34.833346 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:34.833320    5689 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b479cdcf5-7hkgg"]
Nov 01 06:29:34.838015 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:34.837990    5689 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b479cdcf5-7hkgg"]
Nov 01 06:29:36.066278 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:36.066242    5689 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70b65140-62f2-4383-a263-5ca1fc7e9a10" path="/var/lib/kubelet/pods/70b65140-62f2-4383-a263-5ca1fc7e9a10/volumes"
Nov 01 06:29:37.760738 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:37.760707    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:29:39.421005 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:29:39.420969    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:30:00.068819 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:30:00.068777    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 06:30:12.987650 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:30:12.987613    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:30:23.356704 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:30:23.356643    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:30:25.501960 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:30:25.501920    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:30:30.075159 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:30:30.075108    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 06:30:48.402595 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:30:48.402563    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:31:00.068828 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:31:00.068734    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 06:31:00.872536 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:31:00.872506    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:31:03.285483 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:31:03.283548    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-tgfm2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 06:31:03.285483 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:31:03.283653    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tgfm2" podUID="5665e928-b4df-455f-99c1-d466700fee62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 06:31:23.757923 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:31:23.757893    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:31:30.067617 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:31:30.067584    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 06:31:32.998219 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:31:32.998187    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:31:43.689356 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:31:43.689322    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:31:56.201187 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:31:56.201155    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:32:00.073507 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:32:00.073471    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 06:32:08.194145 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:32:08.194113    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:32:24.078903 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:32:24.078865    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:32:27.238206 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:32:27.238166    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:32:30.068870 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:32:30.068836    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 06:32:31.864345 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:32:31.864316    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:32:42.465895 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:32:42.465862    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:32:52.263135 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:32:52.263096    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:33:00.069011 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:33:00.068979    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 06:33:11.475649 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:33:11.475610    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:33:24.429687 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:33:24.429646    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:33:30.068604 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:33:30.068566    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 06:33:35.173061 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:33:35.173023    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:34:00.152979 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:34:00.152709    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 06:34:06.892357 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:34:06.892325    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:34:24.793001 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:34:24.792966    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:34:30.167685 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:34:30.167634    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 06:35:00.069405 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:35:00.069308    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 06:35:03.275583 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:35:03.275548    5689 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58d898bd55-tgfm2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 06:35:03.276182 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:35:03.276147    5689 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58d898bd55-tgfm2" podUID="5665e928-b4df-455f-99c1-d466700fee62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 06:35:25.037722 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:35:25.037685    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:35:30.072989 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:35:30.072949    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 06:35:32.796551 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:35:32.796519    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dc89h"]
Nov 01 06:36:00.073491 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:36:00.073425    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 06:36:03.612858 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:36:03.612817    5689 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-hstph_4959df17-a373-4cef-b555-e342fb4ebbdf/manager/0.log"
Nov 01 06:36:04.774639 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:36:04.774596    5689 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-74b5bc6fff-6xcrb_55552040-c621-4372-89bd-7e41eb45f41e/manager/0.log"
Nov 01 06:36:05.474058 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:36:05.474033    5689 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-pjms7_ff4e020b-0316-4b29-a761-904818e3fb4f/manager/0.log"
Nov 01 06:36:07.412626 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:36:07.412594    5689 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5cdb759fdc-ww7m7_262f6665-9ed2-41fc-9820-b77abf3d654d/maas-api/0.log"
Nov 01 06:36:07.968444 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:36:07.968413    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:36:08.148585 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:36:08.148550    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:36:11.227598 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:36:11.227568    5689 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-g999k_c552fda2-a502-4a75-94cc-dcdf0b7087fb/postgres/0.log"
Nov 01 06:36:14.732362 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:36:14.732332    5689 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-dc89h_817d6a15-8f0e-406f-8dd2-bed3b6323796/limitador/0.log"
Nov 01 06:36:16.191777 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:36:16.191744    5689 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-zpxqq_b54edd8b-ea2a-4035-9dd6-c4628eb73209/discovery/0.log"
Nov 01 06:36:16.555310 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:36:16.555232    5689 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5f68b7bd84-5rtgt_93b41328-171b-43ba-9a18-370d66e0cd5d/kube-auth-proxy/0.log"
Nov 01 06:36:17.533680 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:36:17.533652    5689 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58d898bd55-c6zlb_d1d4ddb1-fd17-4ac1-ada4-f785008a0913/istio-proxy/0.log"
Nov 01 06:36:17.886779 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:36:17.886688    5689 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58d898bd55-cd9pn_ce6f3d72-c7f9-43a9-83ce-40ab705d092a/istio-proxy/0.log"
Nov 01 06:36:18.941504 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:36:18.941471    5689 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58d898bd55-tgfm2_5665e928-b4df-455f-99c1-d466700fee62/istio-proxy/0.log"
Nov 01 06:36:19.273389 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:36:19.273314    5689 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58d898bd55-w2nmc_e6319ad1-4be9-44bb-898a-139e04529f67/istio-proxy/0.log"
Nov 01 06:36:21.784893 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:36:21.784858    5689 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-6h4ll_9c142295-167c-45dc-8287-cce5bb7826c8/main/0.log"
Nov 01 06:36:22.327717 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:36:22.327683    5689 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-d797t_7109ff45-d2c8-42ce-9196-931fc105574c/main/0.log"
Nov 01 06:36:22.555949 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:36:22.555912    5689 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-bpkrz_bffb1fa8-30a6-47ef-8766-1efb36d2d8c4/main/0.log"
Nov 01 06:36:23.035350 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:36:23.035317    5689 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-jd2fn_43840797-117e-46c7-8ad5-6a04151b0b54/main/0.log"
Nov 01 06:36:25.428173 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:36:25.428140    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:36:30.068610 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:36:30.068578    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 06:36:33.899417 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:36:33.899366    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:36:34.079101 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:36:34.079072    5689 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log" limit=1024
Nov 01 06:36:42.827004 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:36:42.826973    5689 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-9pcxf_e856df5a-5c5e-4421-ae7d-80a760bc72e2/global-pull-secret-syncer/0.log"
Nov 01 06:36:45.986346 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:36:45.986311    5689 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-hzr4h_cca47ebc-569a-4558-af41-5e3755605b9d/konnectivity-agent/0.log"
Nov 01 06:36:46.491916 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:36:46.491891    5689 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-hstph_4959df17-a373-4cef-b555-e342fb4ebbdf/manager/0.log"
Nov 01 06:36:47.379634 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:36:47.379601    5689 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-92.us-west-1.compute.internal_0742a37cb8250b02dcdb2d2301dda7a1/haproxy/0.log"
Nov 01 06:36:47.485688 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:36:47.485659    5689 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-74b5bc6fff-6xcrb_55552040-c621-4372-89bd-7e41eb45f41e/manager/0.log"
Nov 01 06:36:48.821197 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:36:48.821164    5689 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-pjms7_ff4e020b-0316-4b29-a761-904818e3fb4f/manager/0.log"
Nov 01 06:36:52.157280 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:36:52.157202    5689 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-hstph_4959df17-a373-4cef-b555-e342fb4ebbdf/manager/0.log"
Nov 01 06:36:53.172422 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:36:53.172386    5689 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-b9nzz/perf-node-gather-daemonset-p8pgn"]
Nov 01 06:36:53.173016 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:36:53.172987    5689 cpu_manager.go:414] "containerMap: removing container" podUID="70b65140-62f2-4383-a263-5ca1fc7e9a10" containerName="maas-api"
Nov 01 06:36:53.173016 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:36:53.173012    5689 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70b65140-62f2-4383-a263-5ca1fc7e9a10" containerName="maas-api"
Nov 01 06:36:53.173173 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:36:53.173098    5689 memory_manager.go:365] "RemoveStaleState removing state" podUID="70b65140-62f2-4383-a263-5ca1fc7e9a10" containerName="maas-api"
Nov 01 06:36:53.176316 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:36:53.176294    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b9nzz/perf-node-gather-daemonset-p8pgn"
Nov 01 06:36:53.267504 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:36:53.267473    5689 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-74b5bc6fff-6xcrb_55552040-c621-4372-89bd-7e41eb45f41e/manager/0.log"
Nov 01 06:36:53.272821 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:36:53.272804    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-b9nzz\"/\"kube-root-ca.crt\""
Nov 01 06:36:53.273087 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:36:53.273071    5689 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-b9nzz\"/\"openshift-service-ca.crt\""
Nov 01 06:36:53.278174 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:36:53.278154    5689 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-b9nzz\"/\"default-dockercfg-87f45\""
Nov 01 06:36:53.279985 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:36:53.279965    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-b9nzz/perf-node-gather-daemonset-p8pgn"]
Nov 01 06:36:53.292081 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:36:53.292056    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/16e45bac-0acc-4ebc-a3a4-6ae17009e121-lib-modules\") pod \"perf-node-gather-daemonset-p8pgn\" (UID: \"16e45bac-0acc-4ebc-a3a4-6ae17009e121\") " pod="openshift-must-gather-b9nzz/perf-node-gather-daemonset-p8pgn"
Nov 01 06:36:53.292177 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:36:53.292161    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/16e45bac-0acc-4ebc-a3a4-6ae17009e121-sys\") pod \"perf-node-gather-daemonset-p8pgn\" (UID: \"16e45bac-0acc-4ebc-a3a4-6ae17009e121\") " pod="openshift-must-gather-b9nzz/perf-node-gather-daemonset-p8pgn"
Nov 01 06:36:53.292244 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:36:53.292188    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/16e45bac-0acc-4ebc-a3a4-6ae17009e121-proc\") pod \"perf-node-gather-daemonset-p8pgn\" (UID: \"16e45bac-0acc-4ebc-a3a4-6ae17009e121\") " pod="openshift-must-gather-b9nzz/perf-node-gather-daemonset-p8pgn"
Nov 01 06:36:53.292304 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:36:53.292277    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/16e45bac-0acc-4ebc-a3a4-6ae17009e121-podres\") pod \"perf-node-gather-daemonset-p8pgn\" (UID: \"16e45bac-0acc-4ebc-a3a4-6ae17009e121\") " pod="openshift-must-gather-b9nzz/perf-node-gather-daemonset-p8pgn"
Nov 01 06:36:53.292362 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:36:53.292349    5689 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjwr6\" (UniqueName: \"kubernetes.io/projected/16e45bac-0acc-4ebc-a3a4-6ae17009e121-kube-api-access-mjwr6\") pod \"perf-node-gather-daemonset-p8pgn\" (UID: \"16e45bac-0acc-4ebc-a3a4-6ae17009e121\") " pod="openshift-must-gather-b9nzz/perf-node-gather-daemonset-p8pgn"
Nov 01 06:36:53.393173 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:36:53.393147    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/16e45bac-0acc-4ebc-a3a4-6ae17009e121-sys\") pod \"perf-node-gather-daemonset-p8pgn\" (UID: \"16e45bac-0acc-4ebc-a3a4-6ae17009e121\") " pod="openshift-must-gather-b9nzz/perf-node-gather-daemonset-p8pgn"
Nov 01 06:36:53.393287 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:36:53.393179    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/16e45bac-0acc-4ebc-a3a4-6ae17009e121-proc\") pod \"perf-node-gather-daemonset-p8pgn\" (UID: \"16e45bac-0acc-4ebc-a3a4-6ae17009e121\") " pod="openshift-must-gather-b9nzz/perf-node-gather-daemonset-p8pgn"
Nov 01 06:36:53.393287 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:36:53.393212    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/16e45bac-0acc-4ebc-a3a4-6ae17009e121-podres\") pod \"perf-node-gather-daemonset-p8pgn\" (UID: \"16e45bac-0acc-4ebc-a3a4-6ae17009e121\") " pod="openshift-must-gather-b9nzz/perf-node-gather-daemonset-p8pgn"
Nov 01 06:36:53.393287 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:36:53.393262    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/16e45bac-0acc-4ebc-a3a4-6ae17009e121-sys\") pod \"perf-node-gather-daemonset-p8pgn\" (UID: \"16e45bac-0acc-4ebc-a3a4-6ae17009e121\") " pod="openshift-must-gather-b9nzz/perf-node-gather-daemonset-p8pgn"
Nov 01 06:36:53.393287 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:36:53.393271    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/16e45bac-0acc-4ebc-a3a4-6ae17009e121-proc\") pod \"perf-node-gather-daemonset-p8pgn\" (UID: \"16e45bac-0acc-4ebc-a3a4-6ae17009e121\") " pod="openshift-must-gather-b9nzz/perf-node-gather-daemonset-p8pgn"
Nov 01 06:36:53.393442 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:36:53.393265    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjwr6\" (UniqueName: \"kubernetes.io/projected/16e45bac-0acc-4ebc-a3a4-6ae17009e121-kube-api-access-mjwr6\") pod \"perf-node-gather-daemonset-p8pgn\" (UID: \"16e45bac-0acc-4ebc-a3a4-6ae17009e121\") " pod="openshift-must-gather-b9nzz/perf-node-gather-daemonset-p8pgn"
Nov 01 06:36:53.393442 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:36:53.393323    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/16e45bac-0acc-4ebc-a3a4-6ae17009e121-podres\") pod \"perf-node-gather-daemonset-p8pgn\" (UID: \"16e45bac-0acc-4ebc-a3a4-6ae17009e121\") " pod="openshift-must-gather-b9nzz/perf-node-gather-daemonset-p8pgn"
Nov 01 06:36:53.393442 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:36:53.393369    5689 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/16e45bac-0acc-4ebc-a3a4-6ae17009e121-lib-modules\") pod \"perf-node-gather-daemonset-p8pgn\" (UID: \"16e45bac-0acc-4ebc-a3a4-6ae17009e121\") " pod="openshift-must-gather-b9nzz/perf-node-gather-daemonset-p8pgn"
Nov 01 06:36:53.393575 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:36:53.393500    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/16e45bac-0acc-4ebc-a3a4-6ae17009e121-lib-modules\") pod \"perf-node-gather-daemonset-p8pgn\" (UID: \"16e45bac-0acc-4ebc-a3a4-6ae17009e121\") " pod="openshift-must-gather-b9nzz/perf-node-gather-daemonset-p8pgn"
Nov 01 06:36:53.454547 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:36:53.454483    5689 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjwr6\" (UniqueName: \"kubernetes.io/projected/16e45bac-0acc-4ebc-a3a4-6ae17009e121-kube-api-access-mjwr6\") pod \"perf-node-gather-daemonset-p8pgn\" (UID: \"16e45bac-0acc-4ebc-a3a4-6ae17009e121\") " pod="openshift-must-gather-b9nzz/perf-node-gather-daemonset-p8pgn"
Nov 01 06:36:53.488200 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:36:53.488179    5689 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b9nzz/perf-node-gather-daemonset-p8pgn"
Nov 01 06:36:53.861382 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:36:53.861190    5689 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-b9nzz/perf-node-gather-daemonset-p8pgn"]
Nov 01 06:36:53.864276 ip-10-0-129-92 kubenswrapper[5689]: W1101 06:36:53.864246    5689 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod16e45bac_0acc_4ebc_a3a4_6ae17009e121.slice/crio-d3d92450dda5a27a2139f668dc09404217945de4c68268b6e6a8b2d225aa8c66 WatchSource:0}: Error finding container d3d92450dda5a27a2139f668dc09404217945de4c68268b6e6a8b2d225aa8c66: Status 404 returned error can't find the container with id d3d92450dda5a27a2139f668dc09404217945de4c68268b6e6a8b2d225aa8c66
Nov 01 06:36:54.232427 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:36:54.232390    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b9nzz/perf-node-gather-daemonset-p8pgn" event={"ID":"16e45bac-0acc-4ebc-a3a4-6ae17009e121","Type":"ContainerStarted","Data":"619cdfb875de501b8932f0a6650c7c49c9f57eea4572af8f32d7e52261e86287"}
Nov 01 06:36:54.232875 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:36:54.232447    5689 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b9nzz/perf-node-gather-daemonset-p8pgn" event={"ID":"16e45bac-0acc-4ebc-a3a4-6ae17009e121","Type":"ContainerStarted","Data":"d3d92450dda5a27a2139f668dc09404217945de4c68268b6e6a8b2d225aa8c66"}
Nov 01 06:36:54.232875 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:36:54.232596    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-b9nzz/perf-node-gather-daemonset-p8pgn"
Nov 01 06:36:54.349039 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:36:54.348982    5689 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-b9nzz/perf-node-gather-daemonset-p8pgn" podStartSLOduration=1.348965209 podStartE2EDuration="1.348965209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 06:36:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:36:54.348590966 +0000 UTC m=+5592.419387687" watchObservedRunningTime="2026-11-01 06:36:54.348965209 +0000 UTC m=+5592.419761931"
Nov 01 06:36:55.006020 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:36:55.005983    5689 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-pjms7_ff4e020b-0316-4b29-a761-904818e3fb4f/manager/0.log"
Nov 01 06:36:55.959150 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:36:55.959115    5689 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-dc89h_817d6a15-8f0e-406f-8dd2-bed3b6323796/limitador/0.log"
Nov 01 06:36:58.455173 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:36:58.455139    5689 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-hstph_4959df17-a373-4cef-b555-e342fb4ebbdf/manager/0.log"
Nov 01 06:36:59.364587 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:36:59.364562    5689 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-74b5bc6fff-6xcrb_55552040-c621-4372-89bd-7e41eb45f41e/manager/0.log"
Nov 01 06:37:00.072242 ip-10-0-129-92 kubenswrapper[5689]: E1101 06:37:00.072209    5689 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkng7" containerName="extractor"
Nov 01 06:37:00.248244 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:37:00.248214    5689 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-b9nzz/perf-node-gather-daemonset-p8pgn"
Nov 01 06:37:00.847469 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:37:00.847429    5689 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-pjms7_ff4e020b-0316-4b29-a761-904818e3fb4f/manager/0.log"
Nov 01 06:37:02.332701 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:37:02.332672    5689 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-5hwhj_805c87a3-63e9-4c13-9c90-c05f73282edc/kube-state-metrics/0.log"
Nov 01 06:37:02.744121 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:37:02.744093    5689 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-5hwhj_805c87a3-63e9-4c13-9c90-c05f73282edc/kube-rbac-proxy-main/0.log"
Nov 01 06:37:03.170407 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:37:03.170368    5689 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-5hwhj_805c87a3-63e9-4c13-9c90-c05f73282edc/kube-rbac-proxy-self/0.log"
Nov 01 06:37:03.588274 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:37:03.588208    5689 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7b6db58fc-994xl_0ddf3d10-7212-4e89-b7a8-e9545a1bcaf6/metrics-server/0.log"
Nov 01 06:37:05.438245 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:37:05.438216    5689 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-zpxqq_b54edd8b-ea2a-4035-9dd6-c4628eb73209/discovery/0.log"
Nov 01 06:37:05.664755 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:37:05.664728    5689 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5f68b7bd84-5rtgt_93b41328-171b-43ba-9a18-370d66e0cd5d/kube-auth-proxy/0.log"
Nov 01 06:37:05.671384 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:37:05.671359    5689 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vh7rl_9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14/node-exporter/0.log"
Nov 01 06:37:06.087647 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:37:06.087619    5689 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vh7rl_9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14/kube-rbac-proxy/0.log"
Nov 01 06:37:06.502922 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:37:06.502889    5689 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vh7rl_9c9e2fb8-c0e2-4cfa-ac00-811559e4ef14/init-textfile/0.log"
Nov 01 06:37:06.540991 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:37:06.540964    5689 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58d898bd55-c6zlb_d1d4ddb1-fd17-4ac1-ada4-f785008a0913/istio-proxy/0.log"
Nov 01 06:37:06.756401 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:37:06.756336    5689 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58d898bd55-cd9pn_ce6f3d72-c7f9-43a9-83ce-40ab705d092a/istio-proxy/0.log"
Nov 01 06:37:07.522058 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:37:07.522034    5689 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58d898bd55-tgfm2_5665e928-b4df-455f-99c1-d466700fee62/istio-proxy/0.log"
Nov 01 06:37:07.738180 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:37:07.738151    5689 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58d898bd55-w2nmc_e6319ad1-4be9-44bb-898a-139e04529f67/istio-proxy/0.log"
Nov 01 06:37:13.173542 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:37:13.173510    5689 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-fbbdp_35391db2-ab6e-4272-8580-9014e24f72c4/prometheus-operator-admission-webhook/0.log"
Nov 01 06:37:13.594108 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:37:13.594041    5689 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-677d79bd4b-jt8p8_c89ad23e-b3d6-417b-8863-1194dc8dc863/telemeter-client/0.log"
Nov 01 06:37:14.007995 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:37:14.007967    5689 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-677d79bd4b-jt8p8_c89ad23e-b3d6-417b-8863-1194dc8dc863/reload/0.log"
Nov 01 06:37:14.422566 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:37:14.422544    5689 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-677d79bd4b-jt8p8_c89ad23e-b3d6-417b-8863-1194dc8dc863/kube-rbac-proxy/0.log"
Nov 01 06:37:14.842997 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:37:14.842926    5689 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-77ddfd4d86-btjkv_80c10aac-5175-46d6-a18b-109169a2461b/thanos-query/0.log"
Nov 01 06:37:15.257853 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:37:15.257827    5689 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-77ddfd4d86-btjkv_80c10aac-5175-46d6-a18b-109169a2461b/kube-rbac-proxy-web/0.log"
Nov 01 06:37:15.670692 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:37:15.670660    5689 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-77ddfd4d86-btjkv_80c10aac-5175-46d6-a18b-109169a2461b/kube-rbac-proxy/0.log"
Nov 01 06:37:15.780818 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:37:15.780791    5689 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-dc89h_817d6a15-8f0e-406f-8dd2-bed3b6323796/limitador/0.log"
Nov 01 06:37:16.084846 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:37:16.084774    5689 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-77ddfd4d86-btjkv_80c10aac-5175-46d6-a18b-109169a2461b/prom-label-proxy/0.log"
Nov 01 06:37:16.498288 ip-10-0-129-92 kubenswrapper[5689]: I1101 06:37:16.498259    5689 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-77ddfd4d86-btjkv_80c10aac-5175-46d6-a18b-109169a2461b/kube-rbac-proxy-rules/0.log"